{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "abb04d61-c097-45a9-9201-e3649cbdc0cc",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "<h1><img width=\"30\" src=\"https://madewithml.com/static/images/rounded_logo.png\">&nbsp;<a href=\"https://madewithml.com/\">Made With ML</a></h1>\n",
    "    <h3>ML for Developers</h3>\n",
    "    Design · Develop · Deploy · Iterate\n",
    "</div>\n",
    "\n",
    "<br>\n",
    "\n",
    "<div align=\"center\">\n",
    "    <a target=\"_blank\" href=\"https://madewithml.com\"><img src=\"https://img.shields.io/badge/Subscribe-40K-brightgreen\"></a>&nbsp;\n",
    "    <a target=\"_blank\" href=\"https://github.com/GokuMohandas/MadeWithML\"><img src=\"https://img.shields.io/github/stars/GokuMohandas/MadeWithML.svg?style=social&label=Star\"></a>&nbsp;\n",
    "    <a target=\"_blank\" href=\"https://www.linkedin.com/in/goku\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n",
    "    <a target=\"_blank\" href=\"https://twitter.com/GokuMohandas\"><img src=\"https://img.shields.io/twitter/follow/GokuMohandas.svg?label=Follow&style=social\"></a>\n",
    "    <br>\n",
    "    🔥&nbsp; Among the <a href=\"https://github.com/GokuMohandas/MadeWithML\" target=\"_blank\">top ML</a> repositories on GitHub\n",
    "</div>\n",
    "\n",
    "<br>\n",
    "<hr>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3012ac29-11de-458c-9023-1a216812f943",
   "metadata": {},
   "source": [
    "# Generative AI"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9dffe7be-4a8f-499a-9531-fa4bbab228ef",
   "metadata": {},
   "source": [
    "In our [Made With ML course](https://madewithml.com/) we will be fine-tuning an LLM for a supervised classification task. The specific class of LLMs we'll be using is called [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)). Bert models are encoder-only models and are the gold-standard for supervised NLP tasks. However, you may be wondering how do all the (much larger) LLM, created for generative applications, fare ([GPT 4](https://openai.com/research/gpt-4), [Falcon 40B](https://huggingface.co/tiiuae/falcon-40b), [Llama 2](https://ai.meta.com/llama/), etc.)?\n",
    "\n",
    "> We chose the smaller BERT model for our course because it's easier to train and fine-tune. However, the workflow for fine-tuning the larger LLMs are quite similar as well. They do require much more compute but Ray abstracts away the scaling complexities involved with that."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d877530b-f3c5-429a-8525-1238c5b8693b",
   "metadata": {},
   "source": [
    "## Set up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2c96931-d511-4c6e-b582-87d24455a11e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install openai==0.27.8 tqdm==4.65.0 -q"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c9fb2cc9",
   "metadata": {},
   "source": [
    "You'll need to first sign up for an [OpenAI account](https://platform.openai.com/signup) and then grab your API key from [here](https://platform.openai.com/account/api-keys)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "953a577e-3cd0-4c6b-81f9-8bc32850214d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import openai\n",
    "openai.api_key = \"YOUR_API_KEY\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3d40cf2d-afa2-41b6-8f03-77b50cc3baca",
   "metadata": {},
   "source": [
    "### Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1790e2f5-6b8b-425c-8842-a2b0ea8f3f07",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b9bfadb-ba49-4f5a-b216-4db14c8888ab",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>created_on</th>\n",
       "      <th>title</th>\n",
       "      <th>description</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>2020-02-20 06:43:18</td>\n",
       "      <td>Comparison between YOLO and RCNN on real world...</td>\n",
       "      <td>Bringing theory to experiment is cool. We can ...</td>\n",
       "      <td>computer-vision</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>2020-02-20 06:47:21</td>\n",
       "      <td>Show, Infer &amp; Tell: Contextual Inference for C...</td>\n",
       "      <td>The beauty of the work lies in the way it arch...</td>\n",
       "      <td>computer-vision</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>2020-02-24 16:24:45</td>\n",
       "      <td>Awesome Graph Classification</td>\n",
       "      <td>A collection of important graph embedding, cla...</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>2020-02-28 23:55:26</td>\n",
       "      <td>Awesome Monte Carlo Tree Search</td>\n",
       "      <td>A curated list of Monte Carlo tree search pape...</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>25</td>\n",
       "      <td>2020-03-07 23:04:31</td>\n",
       "      <td>AttentionWalk</td>\n",
       "      <td>A PyTorch Implementation of \"Watch Your Step: ...</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id           created_on                                              title   \n",
       "0   6  2020-02-20 06:43:18  Comparison between YOLO and RCNN on real world...  \\\n",
       "1   7  2020-02-20 06:47:21  Show, Infer & Tell: Contextual Inference for C...   \n",
       "2   9  2020-02-24 16:24:45                       Awesome Graph Classification   \n",
       "3  15  2020-02-28 23:55:26                    Awesome Monte Carlo Tree Search   \n",
       "4  25  2020-03-07 23:04:31                                      AttentionWalk   \n",
       "\n",
       "                                         description              tag  \n",
       "0  Bringing theory to experiment is cool. We can ...  computer-vision  \n",
       "1  The beauty of the work lies in the way it arch...  computer-vision  \n",
       "2  A collection of important graph embedding, cla...            other  \n",
       "3  A curated list of Monte Carlo tree search pape...            other  \n",
       "4  A PyTorch Implementation of \"Watch Your Step: ...            other  "
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load training data\n",
    "DATASET_LOC = \"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/dataset.csv\"\n",
    "train_df = pd.read_csv(DATASET_LOC)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa5b95d5-d61e-48e4-9100-d9d2fc0d53fa",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['computer-vision', 'other', 'natural-language-processing', 'mlops']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Unique labels\n",
    "tags = train_df.tag.unique().tolist()\n",
    "tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c828129-8248-4e38-93a4-cabb097e7ba5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load inference dataset\n",
    "HOLDOUT_LOC = \"https://raw.githubusercontent.com/GokuMohandas/Made-With-ML/main/datasets/holdout.csv\"\n",
    "test_df = pd.read_csv(HOLDOUT_LOC)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "085dd167-0bee-4167-b1b7-d5797ebda30b",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5686dd79-02bc-4f11-8948-888b58bd504c",
   "metadata": {},
   "source": [
    "We'll define a few utility functions to make the OpenAI request and to store our predictions. While we could perform batch prediction by loading samples until the context length is reached, we'll just perform one at a time since it's not too many data points and we can have fully deterministic behavior (if you insert new data, etc.). We'll also added some reliability in case we overload the endpoints with too many request at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e3c3f44-2c19-4c32-9bc5-e9a7a917d19d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns; sns.set_theme()\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "import time\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4950bdb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I'm an AI with no emotions, just code,\n",
      "But I'm here to help you, lighten your load.\n",
      "So ask me questions, and I'll do my best,\n",
      "To answer in rhymes, and put your mind at rest.\n"
     ]
    }
   ],
   "source": [
    "# Query OpenAI endpoint\n",
    "system_content = \"you only answer in rhymes\"  # system content (behavior)\n",
    "assistant_content = \"\"  # assistant content (context)\n",
    "user_content = \"how are you\"  # user content (message)\n",
    "response = openai.ChatCompletion.create(\n",
    "    model=\"gpt-3.5-turbo-0613\",\n",
    "    messages=[\n",
    "        {\"role\": \"system\", \"content\": system_content},\n",
    "        {\"role\": \"assistant\", \"content\": assistant_content},\n",
    "        {\"role\": \"user\", \"content\": user_content},\n",
    "    ],\n",
    ")\n",
    "print (response.to_dict()[\"choices\"][0].to_dict()[\"message\"][\"content\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "175dddcc",
   "metadata": {},
   "source": [
    "Now let's create a function that can predict tags for a given sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2aae14c-9870-4a27-b5ad-90f339686620",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_tag(model, system_content=\"\", assistant_content=\"\", user_content=\"\"):\n",
    "    try:\n",
    "        # Get response from OpenAI\n",
    "        response = openai.ChatCompletion.create(\n",
    "            model=model,\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": system_content},\n",
    "                {\"role\": \"assistant\", \"content\": assistant_content},\n",
    "                {\"role\": \"user\", \"content\": user_content},\n",
    "            ],\n",
    "        )\n",
    "        predicted_tag = response.to_dict()[\"choices\"][0].to_dict()[\"message\"][\"content\"]\n",
    "        return predicted_tag\n",
    "\n",
    "    except (openai.error.ServiceUnavailableError, openai.error.APIError) as e:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03ee23e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "natural-language-processing\n"
     ]
    }
   ],
   "source": [
    "# Get tag\n",
    "model = \"gpt-3.5-turbo-0613\"\n",
    "system_context = f\"\"\"\n",
    "    You are a NLP prediction service that predicts the label given an input's title and description.\n",
    "    You must choose between one of the following labels for each input: {tags}.\n",
    "    Only respond with the label name and nothing else.\n",
    "    \"\"\"\n",
    "assistant_content = \"\"\n",
    "user_context = \"Transfer learning with transformers: Using transformers for transfer learning on text classification tasks.\"\n",
    "tag = get_tag(model=model, system_content=system_context, assistant_content=assistant_content, user_content=user_context)\n",
    "print (tag)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3087ead2",
   "metadata": {},
   "source": [
    "Next, let's create a function that can predict tags for a list of inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71c43e8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'title': 'Diffusion to Vector',\n",
       "  'description': 'Reference implementation of Diffusion2Vec (Complenet 2018) built on Gensim and NetworkX. '},\n",
       " {'title': 'Graph Wavelet Neural Network',\n",
       "  'description': 'A PyTorch implementation of \"Graph Wavelet Neural Network\" (ICLR 2019) '},\n",
       " {'title': 'Capsule Graph Neural Network',\n",
       "  'description': 'A PyTorch implementation of \"Capsule Graph Neural Network\" (ICLR 2019).'}]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# List of dicts w/ {title, description} (just the first 3 samples for now)\n",
    "samples = test_df[[\"title\", \"description\"]].to_dict(orient=\"records\")[:3]\n",
    "samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9359a91-ac19-48a4-babb-e65d53f39b42",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_predictions(inputs, model, system_content, assistant_content=\"\"):\n",
    "    y_pred = []\n",
    "    for item in tqdm(inputs):\n",
    "        # Convert item dict to string\n",
    "        user_content = str(item)\n",
    "\n",
    "        # Get prediction\n",
    "        predicted_tag = get_tag(\n",
    "            model=model, system_content=system_content,\n",
    "            assistant_content=assistant_content, user_content=user_content)\n",
    "\n",
    "        # If error, try again after pause (repeatedly until success)\n",
    "        while predicted_tag is None:\n",
    "            time.sleep(30)  # could also do exponential backoff\n",
    "            predicted_tag = get_tag(\n",
    "                model=model, system_content=system_content,\n",
    "                assistant_content=assistant_content, user_content=user_content)\n",
    "\n",
    "        # Add to list of predictions\n",
    "        y_pred.append(predicted_tag)\n",
    "\n",
    "    return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fac795e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/3 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [00:01<00:00,  2.04it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['other', 'computer-vision', 'computer-vision']"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get predictions for a list of inputs\n",
    "get_predictions(inputs=samples, model=model, system_content=system_context)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "190e7e72",
   "metadata": {},
   "source": [
    "Next we'll define a function that can clean our predictions in the event that it's not the proper format or has hallucinated a tag outside of our expected tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4cb38a8-44cb-4cea-828c-590f223d4063",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def clean_predictions(y_pred, tags, default=\"other\"):\n",
    "    for i, item in enumerate(y_pred):\n",
    "        if item not in tags:  # hallucinations\n",
    "            y_pred[i] = default\n",
    "        if item.startswith(\"'\") and item.endswith(\"'\"):  # GPT 4 likes to places quotes\n",
    "            y_pred[i] = item[1:-1]\n",
    "    return y_pred"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e6f18bd5",
   "metadata": {},
   "source": [
    "> Open AI has now released [function calling](https://openai.com/blog/function-calling-and-other-api-updates) and [custom instructions](https://openai.com/blog/custom-instructions-for-chatgpt) which is worth exploring to avoid this manual cleaning."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f361ee27",
   "metadata": {},
   "source": [
    "Next, we'll define a function that will plot our ground truth labels and predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de2d0416",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_tag_dist(y_true, y_pred):\n",
    "    # Distribution of tags\n",
    "    true_tag_freq = dict(Counter(y_true))\n",
    "    pred_tag_freq = dict(Counter(y_pred))\n",
    "    df_true = pd.DataFrame({\"tag\": list(true_tag_freq.keys()), \"freq\": list(true_tag_freq.values()), \"source\": \"true\"})\n",
    "    df_pred = pd.DataFrame({\"tag\": list(pred_tag_freq.keys()), \"freq\": list(pred_tag_freq.values()), \"source\": \"pred\"})\n",
    "    df = pd.concat([df_true, df_pred], ignore_index=True)\n",
    "\n",
    "    # Plot\n",
    "    plt.figure(figsize=(10, 3))\n",
    "    plt.title(\"Tag distribution\", fontsize=14)\n",
    "    ax = sns.barplot(x=\"tag\", y=\"freq\", hue=\"source\", data=df)\n",
    "    ax.set_xticklabels(list(true_tag_freq.keys()), rotation=0, fontsize=8)\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1fc0b6ad",
   "metadata": {},
   "source": [
    "And finally, we'll define a function that will combine all the utilities above to predict, clean and plot our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff3c37fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(test_df, model, system_content, assistant_content, tags):\n",
    "    # Predictions\n",
    "    y_test = test_df.tag.to_list()\n",
    "    test_samples = test_df[[\"title\", \"description\"]].to_dict(orient=\"records\")\n",
    "    y_pred = get_predictions(\n",
    "        inputs=test_samples, model=model,\n",
    "        system_content=system_content, assistant_content=assistant_content)\n",
    "    y_pred = clean_predictions(y_pred=y_pred, tags=tags)\n",
    "\n",
    "    # Performance\n",
    "    metrics = precision_recall_fscore_support(y_test, y_pred, average=\"weighted\")\n",
    "    performance = {\"precision\": metrics[0], \"recall\": metrics[1], \"f1\": metrics[2]}\n",
    "    print(json.dumps(performance, indent=2))\n",
    "    plot_tag_dist(y_true=y_test, y_pred=y_pred)\n",
    "    return y_pred, performance"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1fb7ab2a",
   "metadata": {},
   "source": [
    "## Benchmarks"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "83f00073",
   "metadata": {},
   "source": [
    "Now we're ready to start benchmarking our different LLMs with different context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "972fee2f-86e2-445e-92d0-923f5690132a",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = {\"zero_shot\": {}, \"few_shot\": {}}\n",
    "performance = {\"zero_shot\": {}, \"few_shot\": {}}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2029fff2-ae81-4cef-bef5-3cac717d0222",
   "metadata": {},
   "source": [
    "### Zero-shot learning"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "341b9d90",
   "metadata": {},
   "source": [
    "We'll start with zero-shot learning which involves providing the model with the `system_content` that tells it how to behave but no examples of the behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ee4e745-ef56-4b76-8230-fcbe56ac46aa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "system_content = f\"\"\"\n",
    "    You are a NLP prediction service that predicts the label given an input's title and description. \n",
    "    You must choose between one of the following labels for each input: {tags}. \n",
    "    Only respond with the label name and nothing else.\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73780054-afeb-4ce6-8255-51bf91f9f820",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 191/191 [01:26<00:00,  2.21it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"precision\": 0.7919133278407181,\n",
      "  \"recall\": 0.806282722513089,\n",
      "  \"f1\": 0.7807530967691199\n",
      "}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAE9CAYAAAAI8PPbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAUklEQVR4nO3deVxU9f7H8ffMAAIiiKjgnpmKlrjkguWKXivDTMmrldg1Ky23Um9YWi5lmXq1RC1N07qmlktamblkqz/RKG2TNHNLUkllEdlk5vz+8OFcJ1yAQWZGXs/Ho0fOWeb7OcP5zvDm+z1nTIZhGAIAAAAAFJvZ1QUAAAAAgKcjWAEAAACAkwhWAAAAAOAkghUAAAAAOIlgBQAAAABOIlgBAAAAgJMIVgAAAADgJIIVAAAAADiJYAUAAAAATiJYAQAKiI+PV8OGDQv139ixY11drmbNmqWGDRtqx44dkqSjR4+qYcOGGjNmTLGe78SJE8rKyrrqdpdqJzY2Vg0bNlR+fn6x2i5KXWPHjlXDhg11+PDhEm8LAFA0Xq4uAADgfv7xj3+odu3aDstefvllpaamatq0aQ7L/76dO6hUqZKmTZumWrVqFXnfVatWacqUKfroo4/k7+9/zdopibr69u2rtm3bqnLlyte8fQDAlRGsAAAFhIeHKzw83GHZa6+9ptTUVPXs2dNFVRWev79/setMSEgo1GiVs+0U1aXqat68uZo3b14q7QMAroypgAAAAADgJIIVAMBpGzdu1MMPP6w2bdro5ptvVps2bTRkyBD9/PPPBbb98MMP1bt3bzVr1kzt27fXjBkztHLlSodrpK7WVp8+fdSsWTN16NBBs2fPltVqddjmUtc+ZWdn6+WXX9add96piIgItWnTRoMHD9Z3331n3yYqKkofffSRJKlLly6KjY2VdP5apiZNmujLL79U586d1aRJE40aNeqK13Lt2bNHsbGxioiI0G233aZnn31WKSkpDttERUWpQ4cOBfa98HqsWbPmqnX9/Rorq9Wq//73v+rZs6ciIiLUokULDRgwQF9++aVDG2vWrFHDhg21fft2TZs2TZ06ddItt9yiO++8U4sXL77KTwEA8HdMBQQAOGXJkiV6+eWX1aZNGw0bNkze3t76+eeftXbtWu3cuVNbtmxRpUqVJEkLFizQf/7zH91888168skndebMGS1durTQbS1btkyTJk1SgwYNNHLkSGVlZWnZsmXKzs6+6r6jRo3Stm3b9OCDD+rGG2/UyZMn9e677+qhhx7SqlWrFB4ermeffVYLFy7Url279Mwzz6h+/fr2/fPz8zVmzBj1799fFStWVFhY2BXbe+ihh9S2bVvFxcVp7969WrlypRISEvTBBx8oKCio0Mcs6Yp1Xcxms2nYsGHaunWr2rRpo9GjR+vs2bNas2aNHnvsMY0dO1YDBw502GfcuHHy9/fXgAED5OXlpWXLlmnq1KkKCAhQnz59ilQnAJRlBCsAQLFZrVa98cYbatSokRYvXiyLxWJfFxgYqEWLFmnnzp268847deLECcXHx+vmm2/WihUr5OPjI0nq2bOnevTocdW2MjMzNX36dNWvX1/vv/++/Pz8JEm9e/e+6nVOp0+f1tatW3X//fcrLi7OvjwyMlJjx47VTz/9pPDwcHXt2lWffvqpdu3apa5du6pmzZr2bW02m/r376+RI0falx09evSybd57772aMGGC/XH9+vX14osvatGiRRo1atRVj/diV6rrYh9++KG2bt2qe++9V1OnTpXJZJIkDRgwQDExMZo+fbq6dOnicMOR8uXLa/Xq1fafR1RUlLp06aLVq1cTrACgCJgKCAAoNovFoq+++kpvv/22Q6jKysqSt7e3pPOBSJK2bNmivLw8Pfzww/Zf4qXzdxW85557rtrW9u3blZWVpfvuu88eqiSpWrVqio6OvuK+AQEBqlChgjZu3KiVK1fqr7/+knT+5g8XphYWxm233Vao7SRp6NChDo/79eunChUqaNOmTYV+jqL69NNPJUkjRoywhyrp/PEPHjxYVqtVGzdudNjnjjvucPh51KxZU8HBwTp58uQ1qxMArkeMWAEAnOLj46PvvvtOGzZs0MGDB5WcnKxjx47JMAxJsv//4MGDkqS6desWeI569epdtZ0jR45Ikm644YYi7+/j46OpU6fqmWee0fjx4yVJDRo0ULt27dSjRw81btz4qu1LUkhISKG2q1ixYoFboHt7e6tmzZr67bffCvUcxXHkyBH5+/urRo0aBdZdmD7491G2KlWqFNjWx8dHNpvt2hQJANcpRqwAAE4ZPXq0/vWvf+m7775T7dq1FRsbq7feekvPP/+8w3Z5eXmS5DA6coGvr2+h28vNzS2w7EJ4u5KuXbvqq6++Unx8vPr27au8vDy99dZb6t27t955551CtX3xqNyVXDxa9Pc6C/Mcxf1y4Su9DheC0t9ff7OZXwUAoCQwYgUAKLbExER9/PHHuuuuuzRr1iyHQLF7926HbS+MVB04cKDAzRcOHDhw1bbq1Klz2W0PHTp0xX0zMzO1d+9e1axZU926dVO3bt0kSUlJSRowYIDmzp2rAQMGXLWGwkpPT1dGRoYCAwPty/Ly8vTHH3/Yj0M6H9Qu9Z1ZxZ2GV7t2bR04cEDJyckFRq0ujJRVr169WM8NALgy/kwFACi2tLQ0SeenmV0cqk6fPq1Vq1ZJ+t/oS7du3eTl5aWlS5fq3Llz9m1TUlLstxK/kttvv11BQUF69913lZGRYV9+6tQprVu37or77t27Vw888IDmzZvnsLx+/fqqUKGCvLz+93fGCyNKhRkFuxybzaZly5Y5LHv77bd19uxZ3XXXXfZlVatWVVpamsP0vNzcXPu1UhcrTF133HGHJGn27NkO2509e1ZvvvmmLBaLunbtWryDAgBcESNWAIBia9GihSpWrKg333xTOTk5ql27to4eParVq1frzJkzkmT/f40aNfT4448rPj5e999/v6Kjo+23S78wanO5KXSS5Ofnp4kTJ2r06NHq3bu3+vbtK8MwtGzZMvuNMq5U52233aYVK1YoIyNDrVu3ltVq1YYNG5ScnOxwp8AL11EtXLhQ7du3L1YQ8fPz0/z583X06FE1adJEu3bt0gcffKCbb75ZDz/8sH27e++9V4mJiRo0aJAefPBB2Ww2rV69+pLhqTB19ezZU59++qnWrl2rY8eOqUuXLsrOztbq1at15MgRjRkzRrVq1Sry8QAAro5gBQAotkqVKumtt97SzJkz9f777ysvL0+hoaG64447NHDgQN155536+uuv9dhjj0mShg0bpsqVK2vp0qWaMWOGgoODFRMTo9zcXC1evPiS119drHv37goODtbcuXM1b948+fr6qkePHqpTp44mT5582f1MJpPi4+P11ltvacOGDfriiy8kSeHh4ZoxY4bD7d779++v77//XqtXr1ZCQkKxglVgYKBeffVVTZ06VevWrVNQUJAeeughjRgxwuF6sj59+igrK0vLly/XtGnTVLlyZfXs2VMdOnTQAw884PCchanLYrFo3rx5evvtt7V27VrNmDFDfn5+atKkiZ577rlLfhkxAKBkmAxn5joAAFBIWVlZslqtqlChQoF1zz33nN5//3199tlnl/2OJgAA3BnXWAEASsVvv/2mli1bas6cOQ7Lz5w5o88//1xVqlS55G3CAQDwBEwFBACUiltuuUUNGzbUG2+8odOnT6tRo0ZKS0vTmjVrdOrUKf3nP/+54jVWAAC4M6YCAgBKzenTp7Vw4UJt2bJFx48fl5+fnyIiIvTII4+oTZs2ri4PAIBiI1gBAAAAgJO4xgoAAAAAnESwAgAAAAAnuVWwmj9/vmJjYx2WJSUlqX///mrWrJmioqL0zjvvOKy32WyaPXu22rdvr2bNmunRRx/VH3/8UZplAwAAACjj3OaugO+++65effVVtWzZ0r4sNTVVAwcOVFRUlCZNmqTdu3dr0qRJKl++vGJiYiRJ8+bN07JlyzR16lSFhYVp+vTpeuSRR/TRRx9d9YsmL8cwDNlsXHoGAAAAlGVms6nQd6x1ebA6ceKEJkyYoB07duiGG25wWPf+++/L29tbkydPlpeXl+rVq6fDhw9rwYIFiomJUV5ent566y2NGTNGnTp1kiTNmjVL7du316ZNmxQdHV2smmw2Q6dPn3XyyAAAAAB4skqVystiKVywcvlUwF9++UXe3t768MMP1bRpU4d1iYmJat26tby8/pf/IiMjdejQIZ08eVK//vqrzp49q7Zt29rXBwYGqnHjxvr2229L7RgAAAAAlG0uH7GKiopSVFTUJdcdP35cDRo0cFhWtWpVSdKxY8d0/PhxSVK1atUKbHNhXXF5ebk8cwIAAADwEC4PVleSk5NT4DqpcuXKSZJyc3OVnZ0tSZfcJj09vdjtms0mBQeXL/b+AAAAAMoWtw5Wvr6+ysvLc1iWm5srSfL395evr68kKS8vz/7vC9v4+fkVu12bzVBGRlax9wcAAADg+QID/WSxFG4mm1sHq7CwMKWkpDgsu/A4NDRU+fn59mW1a9d22KZhw4ZOtZ2fb3NqfwAAAABlh1sHq1atWmnFihWyWq2yWCySpISEBNWtW1chISGqUKGCAgICtGPHDnuwysjI0J49e9S/f39Xlg4AAABcMzabTVZrvqvL8HgWi5fM5pK5t4JbB6uYmBgtXLhQ48aN0yOPPKIff/xRS5Ys0aRJkySdv7aqf//+mjFjhipVqqQaNWpo+vTpCgsLU7du3VxcPQAAAFCyDMNQRsZpZWdnurqU64afX4ACAysV+vuqLsetg1VISIgWLlyoKVOmqFevXqpSpYqefvpp9erVy77NiBEjlJ+fr/HjxysnJ0etWrXSokWL5O3t7cLKAQAAgJJ3IVQFBATLx6ec02GgLDMMQ3l5ucrMTJUkBQWFOPV8JsMwjJIo7Hpitdr4gmAALmc2m2Q284FZFDabIZuNjzUA1yebzaqUlKMKCAhWQECgq8u5bmRmZigzM1VVq9YqMC3w/BcEXwc3rwCAsur81z74yWy2uLoUj2KzWZWamk24AnBdslqtkiQfn3IuruT6cuH1tFrzZTb7XGXryyNYAYAbOj9aZdHBj99U9qljri7HI/iFVFPd6EdlNpsIVgCua0z/K1kl9XoSrADAjWWfOqbsE0dcXQYAwM25cvo407DPI1gBAAAAHsxsNqliRf9CXwtU0qxWm9LSsooUro4fP66ff/5BXbvecQ0rK10EKwAAAMCDmc0mWSxmzV2+Tckp6aXado2qQRp6/+1FnoY9ZcoEhYVVI1gBAAAAcC/JKek6lJzq6jIK5Xq8MTnBCgAAAECpGTbsMe3e/b127/5eu3Z9J0nq1KmLEhK2KTX1tF58cZoWLZqvatWqa9y4iQ77Xbzs0KGDmjNnln74YZf8/f3VokUrDRv2pEJCKrvgqCTXTMQEAAAAUCa99NJ03XJLhKKi/qE333xHkrRmzfsaOXKM/vOfeN18c5OrPsfJk39p6NBHVLNmbS1c+F+98sqrOns2U0OGPKzs7OxrfQiXRLACAAAAUGoCA4Pk5eWlcuXKKTg4WJIUGXm7WrVqo/DwxvLxufp3SX3wwSpVqRKqJ58cozp1blB4eCNNnjxVp0+f0uefb7nWh3BJTAUEAAAA4FI1a9Yq0vb79v2qgwd/1z/+0d5heV5eng4dOliSpRUawQoAAACAS5UrV+6q21itVvu/bTZDLVq01OjRYwtsFxBQoURrKyymAgIAAAAoVSbTlb/M2NvbW2fPnrU/ttls+vPPo/bHN95YT4cPH1LVqqGqWbOWataspcDAQM2e/R8dOLD/mtV9JQQrAAAAAKXKz89fx479qZSUE5dcf8stEfr22x1KSPg/HT36h2bNmq4zZzLt63v1uk+ZmZmaPHm8fvttn377bZ+ef/4ZJSXtUd269UrrMBwwFRAAAAC4DtSoGuQxbd57b4ymTJmghx66X35+fgXW9+v3oJKTj+q558bKx8dbd9/dU127drN//1X16jU0Z858vfHGHD3xxCBZLBY1adJUs2e/Yb8hRmkzGdfjt3M5yWq16fTps1ffEACuES8vs4KDy2vP25OVfeKIq8vxCH6htdX4oeeVmnpW+fk2V5cDACXu3Lk8nTp1TCEh1eTt/b8755nNJlWs6C+LxTWT0axWm9LSsmSzeWasuNzrKkmVKpUv9OvKiBUAAADgwWw2Q2lpWTKbr3zd0rVs31NDVUkiWAEAAAAejnDjety8AgAAAACcRLACAAAAACcRrAAAAADASQQrAAAAAHASwQoAAAAAnESwAgAAAAAnEawAAAAAwEl8jxUAAADg4cxmE18Q7GIEKwAAAMCDmc0mBQf7yWy2uKR9m82q1NRstw1X993XQ3fdFa1BgwZf03YIVgAAAIAHOz9aZdHBj99U9qljpdq2X0g11Y1+VGazyW2DVWkhWAEAAADXgexTx5R94oiryyizCFYAAAAASlW7di311FNPa+PGT7R//z7VrFlLjz32hNq16yhJWrRovnbt+k4hISHavv3/dNddd+upp57WTz/9oDfemKOkpD2qWLGibr+9g4YMGary5QMkSZmZmXr11en65psv5eXlpf79/1Vqx8RdAQEAAACUujfemKM77uiuJUuWqW3bdnr22X/rp59+sK/fvft7VapUWYsXv6v77uun/ft/05NPPqE2bdrq7beXa8KEKdq7N0lPPTVMhnF+GuLzz49VUtIveuWVWZo1a662b9+m48dLZ3okwQoAAABAqevePVoxMf9U7do36PHHhys8vLFWrXrPYZtBgwarRo2aqlWrtpYvf0etW0dqwICHVatWbTVt2kwTJ07Rnj0/a9eu73TkyCHt3Jmgp556Wk2bNlf9+g01YcKL8vHxKZXjYSogAAAAgFLXokVLh8dNmkRo584E++Pg4EoKCAiwP967d6+OHj2if/yjfYHnOnz4kNLT0yRJjRo1ti+vVClE1avXKOHKL41gBQAAAKDUWSyOUcRqtTncMr5cuXIO6w3Dpm7d7tKAAQ8XeK6KFYOVmLhDkgrcnfDv7VwrHjEVMD8/X6+99po6d+6s5s2b68EHH9Tu3bvt65OSktS/f381a9ZMUVFReuedd1xXLAAAAICr+vXXPQ6Pf/75RzVsGH7Z7evWraeDBw+oZs1a9v+sVqtmz56plJTjql+/oSQ5XKd15swZJSf/cW0O4G88Ili9/vrrWrlypV544QWtXbtWdevW1SOPPKKUlBSlpqZq4MCBql27tlavXq2hQ4dqxowZWr16tavLBgAAAHAZ77+/XJs2faojRw5rzpxXtX//Pv3znw9cdvt+/fpr375f9Z//vKJDhw7q559/1MSJz+ro0SOqVauOatSoqc6du2rWrGn69tsdOnBgv1544XmdO3euVI7HI6YCbtmyRdHR0WrXrp0kaezYsVq5cqV2796tgwcPytvbW5MnT5aXl5fq1aunw4cPa8GCBYqJiXFx5QAAAEDp8Aup5lFt3ntvb73//jIdOLBf9erV18yZc3TTTfUvu/0ttzTRzJlztHDh63r44f7y9/fTrbe20tChT8rb21uSNH78RM2Z85omTHhWNptNPXv2VlpaarFrLAqPCFYhISH6/PPP1b9/f1WrVk3vvfeefHx8FB4erpUrV6p169by8vrfoURGRmr+/Pk6efKkKleu7MLKAQAAgGvLZjNks1lVN/pRF7VvLXBdU2HccMONeuKJkZdcN2jQYA0aNLjA8ltvbaVbb2112ecsV85Xo0fHafTouCLX4yyPCFbjxo3TyJEj1aVLF1ksFpnNZsXHx6t27do6fvy4GjRo4LB91apVJUnHjh0jWAEAAOC6ZrMZSk3Nltlscln7xQlW1xuPCFb79+9XhQoVNHfuXIWGhmrlypUaM2aMli5dqpycnAL3pr9wB5Hc3Nxit+nl5RGXnwG4TlksvAcVF68dgOuVzXb54ES4cZ7FYnIqA7h9sDp27JhGjx6tJUuWqGXL8/e6b9Kkifbv36/4+Hj5+voqLy/PYZ8Lgcrf379YbZrNJgUHl3eucACASwQG+rm6BAC4JnJyLDp50ux0AHAHCQnfu7oEO5vNJLPZrKAgf/n6+hb7edw+WP3www86d+6cmjRp4rC8adOm+uqrr1S9enWlpKQ4rLvwODQ0tFht2myGMjKyilcwAJQAi8VMQCimjIxsWa02V5cBACUuLy9XNptNVquh/Hze50qK1WrIZrMpPT1L2dlWh3WBgX6Fngnh9sEqLCxM0vlvWo6IiLAv37dvn2644QY1bdpUK1askNVqlcVy/gvFEhISVLduXYWEhBS7XU5WAPBMVquN93AA1yWrlal+15KzgdXtg1VERIRuvfVWxcXFacKECQoLC9PatWu1fft2LV++XDVr1tTChQs1btw4PfLII/rxxx+1ZMkSTZo0ydWlAwBgZzabXHZhuSfjuhGgIMOgT5Skkno93T5Ymc1mvf7663r11Vf1zDPPKD09XQ0aNNCSJUvUtGlTSdLChQs1ZcoU9erVS1WqVNHTTz+tXr16ubhyAADOM5tNqljRnxtrFIPValNaWhbhCpDss7Py8nLl41POxdVcP/Lyzt+fwWJxLhqZDCJvAVarTadPn3V1GQDKMC8vs4KDy2vP25OVfeKIq8vxCH6htdX4oeeVmnrW7aYCXvh5zl2+Tckp6a4ux2PUqBqkofff7pY/U8BV0tNPKTs7UwEBwfLxKSeTiZHw4jIMQ3l5ucrMTJWfX4CCggpeRlSpUvnr5xorAACuF8kp6TqUnOrqMgB4sMDASpKkzEzeS0qKn1+A/XV1BsEKAAAA8BAmk0lBQSGqUCFYVmu+q8vxeBaLl8zmkpmmTbACAAAAPIzZbJbZ7OPqMnARrqIFAAAAACcRrAAAAADASQQrAAAAAHASwQoAAAAAnESwAgAAAAAnEawAAAAAwEkEKwAAAABwEsEKAAAAAJxEsAIAAAAAJxGsAAAAAMBJXq4uAMD1z2w2yWw2uboMj2Kx8Hcv4AL6Q9HYbIZsNsPVZQBlDsEKwDVlNptUsaI/vxgBKLKgCr4ybDYFBvq5uhSPYrNZlZqaTbgCShnBCsA1ZTabZLGYNXf5NiWnpLu6HI/RtGF19b2zmavLAFyqvK+PTGazDn78prJPHXN1OR7BL6Sa6kY/KrPZRLACShnBCkCpSE5J16HkVFeX4TGqVwl0dQmA28g+dUzZJ464ugwAuCLm5gAAAACAkwhWAAAAAOAkghUAAAAAOIlgBQAAAABOIlgBAAAAgJMIVgAAAADgJIIVAAAAADiJYAUAAAAATiJYAQAAAICTCFYAAAAA4CSCFQAAAAA4iWAFAAAAAE4iWAEAAACAkwhWAAAAAOAkghUAAAAAOIlgBQAAAABOIlgBAAAAgJM8JlitXbtW3bt3V5MmTXT33Xdrw4YN9nVHjx7V4MGD1aJFC7Vr106vvvqqrFarC6sFAAAAUJZ4RLBat26dxo0bpwcffFDr169XdHS0Ro0apV27duncuXMaNGiQJGnFihWaOHGili9frrlz57q4agAAAABlhZerC7gawzD02muvacCAAXrwwQclSY8//rgSExO1c+dOJScn688//9T777+voKAgNWjQQKdOndK0adM0ZMgQ+fj4uPgIAAAAAFzv3H7E6uDBg0pOTlaPHj0cli9atEiDBw9WYmKibr75ZgUFBdnXRUZGKjMzU0lJSaVdLgAAAIAyyO1HrA4ePChJysrK0qBBg7Rnzx7VrFlTjz/+uKKionT8+HGFhYU57FO1alVJ0rFjx9S0adNitevl5faZE/AIFgt9CaXLHc85d6wJ1zfOOaD0uX2wyszMlCTFxcVp2LBhGjNmjDZu3KgnnnhCixcvVk5OjgIDAx32KVeunCQpNze3WG2azSYFB5d3rnAAgEsEBvq5ugTA5egHQOlz+2Dl7e0tSRo0aJB69eolSWrUqJH27NmjxYsXy9fXV3l5eQ77XAhU/v7+xWrTZjOUkZHlRNUALrBYzHzAo1RlZGTLarW5ugwH9AOUNnfsB4AnCgz0K/QIsNsHq9DQUElSgwYNHJbfdNNN+uKLL9S6dWvt27fPYV1KSorDvsWRn8+bEQB4IqvVxns4yjz6AVD63H4C7s0336zy5cvrhx9+cFi+b98+1a5dW61atdKePXvsUwYlKSEhQeXLl1d4eHhplwsAAACgDHL7YOXr66tHHnlEc+fO1ccff6wjR47o9ddf17Zt2zRw4EB17dpVVapU0ZNPPqlff/1VW7Zs0cyZM/Xwww9zq3UAAAAApcLtpwJK0hNPPCE/Pz/NmjVLJ06cUL169RQfH682bdpIkhYuXKhJkybpn//8p4KCgvTAAw/oiSeecHHVAAAAAMoKjwhWkjRw4EANHDjwkuvq1Kmjt956q5QrAgAAAIDz3H4qIAAAAAC4O4IVAAAAADipyFMBn3nmmUJvazKZ9NJLLxW1CQAAAADwKEUOVsePH9eePXuUnp6uGjVqKDQ0VGlpaTp8+LAMw1BYWJh9W5PJVKLFAgAAAIA7KnKw6t69u3777TctW7ZMLVq0sC8/cOCAHn/8cT3wwAN66KGHSrRIAAAAAHBnRb7G6o033tCYMWMcQpUk3XjjjXryySe1aNGiEisOAAAAADxBkYPV6dOnFRQUdOknM5t15swZp4sCAAAAAE9S5GDVtGlTzZkzR6mpqQ7LU1JSFB8fr3bt2pVYcQAAAADgCYp8jdXYsWPVv39/RUVFqXnz5goODtapU6e0a9cuhYSE6Nlnn70WdQIAAACA2yryiFV4eLjWr1+vfv36KTMzUz///LNycnL08MMPa82aNapWrdq1qBMAAAAA3FaRR6wkKTQ0VHFxcSVdCwAAAAB4pGIFq7y8PK1atUr/93//p7/++ksvvfSSdu7cqZtvvlkRERElXSMAAAAAuLVi3RUwJiZGU6ZM0eHDh/Xjjz8qJydHX3zxhWJjY7Vr165rUScAAAAAuK0iB6tp06bp7Nmz+uSTT/TBBx/IMAxJ0uzZs9WkSRPNnj27xIsEAAAAAHdW5GD1+eefa+TIkapTp45MJpN9ebly5fTwww/rl19+KdECAQAAAMDdFTlY5ebmqmLFipdcZ7FYdO7cOWdrAgAAAACPUuRg1aRJEy1btuyS6z766CPdcsstThcFAAAAAJ6kyHcFHDlypP71r3+pZ8+e6tixo0wmkz7++GPFx8frm2++0cKFC69FnQAAAADgtoo8YtWyZUstXrxYfn5+WrhwoQzD0JIlS/TXX39p/vz5ioyMvBZ1AgAAAIDbKvKI1fbt29W8eXOtWLFCOTk5Sk9PV0BAgMqXL38t6gMAAAAAt1fkEavhw4dr06ZNkiRfX1+FhoYSqgAAAACUaUUOVoGBgfL19b0WtQAAAACARyryVMDBgwfrxRdf1MGDBxUeHi5/f/8C27Rq1apEigMAAAAAT1CoYJWbm6ty5cpJkiZMmCBJmjVrliQ5fEmwYRgymUxKSkoq6ToBAAAAwG0VKlhFRUVpzpw5at68uVq1aqU+ffooLCzsWtcGAAAAAB6hUMHqzJkzSklJkSQlJibq3//+tyIiIq5pYQAAAADgKQoVrJo0aaLRo0frlVdekWEYGjp0qHx8fC65rclk0pYtW0q0SAAAAABwZ4UKVjNnztSSJUuUlpamDz74QI0bN1alSpWudW0AAAAA4BEKFaxCQ0MVFxcnSdqxY4eeeuophYeHX9PCAAAAAMBTFPl261u3br0WdQAAAACAxyryFwQDAAAAABwRrAAAAADASQQrAAAAAHCSRwWrgwcPqnnz5lqzZo19WVJSkvr3769mzZopKipK77zzjgsrBAAAAFAWeUywOnfunMaMGaOsrCz7stTUVA0cOFC1a9fW6tWrNXToUM2YMUOrV692YaUAAAAAypoi3xXQVeLj4xUQEOCw7P3335e3t7cmT54sLy8v1atXT4cPH9aCBQsUExPjokoBAAAAlDUeMWL17bff6r333tPUqVMdlicmJqp169by8vpfPoyMjNShQ4d08uTJ0i4TAAAAQBnl9iNWGRkZevrppzV+/HhVq1bNYd3x48fVoEEDh2VVq1aVJB07dkyVK1cudrteXh6ROQG3Z7HQl1C63PGcc8eacH3jnANKn9sHq4kTJ6p58+bq0aNHgXU5OTny8fFxWFauXDlJUm5ubrHbNJtNCg4uX+z9AQCuExjo5+oSAJejHwClz62D1dq1a5WYmKiPPvrokut9fX2Vl5fnsOxCoPL39y92uzaboYyMrKtvCOCqLBYzH/AoVRkZ2bJaba4uwwH9AKXNHfsB4IkCA/0KPQLs1sFq9erVOnXqlDp16uSwfMKECfrkk08UFhamlJQUh3UXHoeGhjrVdn4+b0YA4ImsVhvv4Sjz6AdA6XPrYDVjxgzl5OQ4LOvWrZtGjBihe+65R+vWrdOKFStktVplsVgkSQkJCapbt65CQkJcUTIAAACAMsitr2wMDQ1VnTp1HP6TpJCQEIWGhiomJkaZmZkaN26c9u/frzVr1mjJkiUaPHiwiysHAAAAUJa4dbC6mpCQEC1cuFAHDx5Ur169NGfOHD399NPq1auXq0sDAAAAUIa49VTAS9m7d6/D44iICL333nsuqgYAAAAAPHzECgAAAADcAcEKAAAAAJxEsAIAAAAAJxGsAAAAAMBJBCsAAAAAcBLBCgAAAACcRLACAAAAACcRrAAAAADASQQrAAAAAHASwQoAAAAAnESwAgAAAAAnEawAAAAAwEkEKwAAAABwEsEKAAAAAJxEsAIAAAAAJxGsAAAAAMBJBCsAAAAAcBLBCgAAAACcRLACAAAAACcRrAAAAADASQQrAAAAAHASwQoAAAAAnESwAgAAAAAnEawAAAAAwEleri6gLDCbTTKbTa4uw6PYbIZsNsPVZQAAAACFQrC6xsxmkypW9JfFwuBgUVitNqWlZRGuAAAA4BEIVteY2WySxWLW3OXblJyS7upyPEKNqkEaev/tMptNBCsAAAB4BIJVKUlOSdeh5FRXlwEAAADgGmB+GgAAAAA4iWAFAAAAAE4iWAEAAACAkwhWAAAAAOAkghUAAAAAOMkjglVaWpqef/55dejQQS1atND999+vxMRE+/rt27erd+/eatq0qe68806tX7/ehdUCAAAAKGs8IliNGjVKu3bt0syZM7V69Wo1atRIgwYN0oEDB/T7779r8ODBat++vdasWaM+ffro6aef1vbt211dNgAAAIAywu2/x+rw4cPatm2bli1bpltvvVWS9Nxzz+nrr7/WRx99pFOnTqlhw4Z66qmnJEn16tXTnj17tHDhQrVt29aVpQMAAAAoI9x+xCo4OFgLFixQkyZN7MtMJpNMJpMyMjKUmJhYIEBFRkbqu+++k2EYpV0uAAAAgDLI7YNVYGCgOnbsKB8fH/uyjRs36vDhw2rfvr2OHz+usLAwh32qVq2q7Oxspaamlna5AAAAAMogt58K+Hfff/+9nnnmGXXr1k2dOnVSTk6OQ+iSZH+cl5dX7Ha8vEomc1osbp9d3Rav3fWBnyNKmzuec+5YE65vnHNA6fOoYLVlyxaNGTNGLVq00IwZMyRJ5cqVKxCgLjz28/MrVjtms0nBweWdKxZOCwws3s8PQNnGewdAPwBcwWOC1dKlSzVlyhTdeeedeuWVV+yjUtWqVVNKSorDtikpKfL391eFChWK1ZbNZigjI8vpmqXzfzHiza14MjKyZbXaXF0GnEQfQGlzx/cO+gFKmzv2A8ATBQb6FXoE2COC1bJly/TCCy8oNjZW48aNk8lksq9r2bKldu7c6bB9QkKCWrRoIbO5+MPg+fm8Gbma1Wrj5wCgyHjvAOgHgCu4fbA6ePCgXnrpJf3jH//Q4MGDdfLkSfs6X19fxcbGqlevXpoxY4Z69eqlL7/8Up9++qkWLlzowqoBAAAAlCVuH6w2btyoc+fOafPmzdq8ebPDul69emnq1KmaN2+epk+frrfffls1a9bU9OnT+Q4rAAAAAKXG7YPVkCFDNGTIkCtu06FDB3Xo0KGUKgIAAAAAR9yLEwAAAACcRLACAAAAACe5/VRAAAAAXB/MZpPMZtPVN4SdzWbIZjNcXQYKgWAFAACAa85sNqliRf9CfycQzrNabUpLyyJceQCCFQAAAK45s9kki8Wsucu3KTkl3dXleIQaVYM09P7bZTabCFYegGAFAACAUpOckq5DyamuLgMocQQrAAAAwI0xfbLoXHFtGsEKAAAAcENBFXxl2GwKDPRzdSkex2azKjU1u1TDFcEKAAAAcEPlfX1kMpt18OM3lX3qmKvL8Rh+IdVUN/rRUr82jWAFAAAAuLHsU8eUfeKIq8vAVTBhEwAAAACcRLACAAAAACcRrAAAAADASQQrAAAAAHASwQoAAAAAnESwAgAAAAAnEawAAAAAwEkEKwAAAABwEl8QDLdlsZD7i8pmM0r1G8YBAABwHsEKbieogq8Mm02BgX6uLsXj2GxWpaZmE64AAABKGcEKbqe8r49MZrMOfvymsk8dc3U5HsMvpJrqRj8qs9lEsAIAAChlBCu4rexTx5R94oirywAAAACuiotYAAAAAMBJBCsAAAAAcBLBCgAAAACcRLACAAAAACcRrAAAAADASQQrAAAAAHASwQoAAAAAnESwAgAAAAAnEawAAAAAwEkEKwAAAABwEsEKAAAAAJxEsAIAAAAAJ10Xwcpms2n27Nlq3769mjVrpkcffVR//PGHq8sCAAAAUEZcF8Fq3rx5WrZsmV544QWtWLFCNptNjzzyiPLy8lxdGgAAAIAywOODVV5ent566y2NGDFCnTp1Unh4uGbNmqXjx49r06ZNri4PAAAAQBng8cHq119/1dmzZ9W2bVv7ssDAQDVu3FjffvutCysDAAAAUFaYDMMwXF2EMzZt2qThw4frhx9+kK+vr335yJEjlZOTo/nz5xf5OQ3DkM1WMi+LySSZzWalZ+bIarWVyHNe73y8LQrwL6dzZzNk2KyuLsdjmMwWeZcPlM1mkzv1avpA8dAPis5d+4BEPygu+kHR0Q+uL/SB4inJfmA2m2QymQq1rZdzTbledna2JMnHx8dhebly5ZSenl6s5zSZTLJYCvcCFlZQgO/VN4ID7/KBri7BI5nN7jkQTR8oHvpB0blrH5DoB8VFPyg6+sH1hT5QPKXdD9y31xXShVGqv9+oIjc3V35+fq4oCQAAAEAZ4/HBqlq1apKklJQUh+UpKSkKDQ11RUkAAAAAyhiPD1bh4eEKCAjQjh077MsyMjK0Z88etWrVyoWVAQAAACgrPP4aKx8fH/Xv318zZsxQpUqVVKNGDU2fPl1hYWHq1q2bq8sDAAAAUAZ4fLCSpBEjRig/P1/jx49XTk6OWrVqpUWLFsnb29vVpQEAAAAoAzz+dusAAAAA4Goef40VAAAAALgawQoAAAAAnESwAgAAAAAnEawAAAAAwEkEKwAAAABwEsEKAAAAAJxEsAIAAAAAJxGs4FKxsbH2fzds2NCFlQCuM3v2bCUmJpb48y5fvlzLly8v9nq4t5I8b8aOHas1a9YUWL5mzRqNHTu2RNrApfXs2dPVJcCN8LuQZ/NydQEo23bu3OnqEgCX+/bbb9WmTZsSf97777/fqfVwb9fqvEHpWrdunatLAFBCCFYoNW+88YY+/PBDWSwW3X777crOzpYk9e7d2/6X0kmTJmnXrl3Kzc3VK6+8ooiICB05ckQTJ05UamqqfHx8FBcXpxYtWmjs2LFKTU3VkSNHNHLkSN15552uPDx4sJkzZ2rjxo2yWCzq2bOnunXrpueff15paWny9/fXuHHjFBERobFjx8rX11e7d+9WWlqannrqKW3ZskVJSUnq3Lmzxo0bpzVr1mjjxo3KzMxUSkqKOnbsqHHjxik5OVkDBgzQ1q1bJZ0fCdi5c6datWqln3/+WePHj9fs2bNVvnz5Ip/vU6dOVaVKlfTYY49JkuLi4tS6dWv9+eefkqShQ4fa+5bFYlFUVJSGDx+u+Ph4SdLw4cP1+eef69VXX5XNZlOtWrU0efJkVa5cWVFRUerZs6e2bdumtLQ0Pffcc2rfvn0p/4TKhh07duj1119XhQoV9PvvvyssLEwzZ87Uxx9/rLVr1yonJ0fS+fP1p59+cjhvXnrpJQ0bNswetBo2bKi9e/cqPj5eu3fv1vHjx9WnTx81btxYM2fOVG5urtLT0zV69Gjdfffdhapvw4YNWrx4sXJycpSTk6PJkycrMjJSsbGxatq0qRITE5WSkqLhw4erV69eyszM1LPPPqvffvtNVapUkclk0hNPPCFJmjNnjv773/9KksN5uHTp0gLH2qBBAyUmJmry5Mkym81q2bKlvvzyS23evFmnT5/W888/bz/Xhw0bpqioqAK1x8bGqm7duvr555+VnZ2tsWPHqmPHjgVen44dO16y7x87dkzPPPOMTp48KR8fH02cOFERERFat26d3n77bVmtVt10002aNGmS/P39L9nfdu/erRdffFGGYahcuXJ68cUXdeONNzr8rE6cOKEjR44oOTnZXoskvfbaa1q/fr0qVKigevXqqVatWho+fHhxTzW4yI4dOzR37lx5eXnp0KFDat++vUJDQ7VlyxbZbDYtWLDAvm12drbGjx+vvXv3ymQyadCgQbr33nsv+xmTm5urp59+WkeOHJHJZFLfvn3Vr18/Fx5tGWUApeCLL74wYmJijKysLOPcuXPGkCFDjKVLlxoNGjSwb9OgQQNj/fr1hmEYxttvv20MHz7cMAzD6Nevn/Hjjz8ahmEYhw8fNjp37mycO3fOiIuLM0aPHl36B4PrysaNG42+ffsaOTk5Rk5OjnHfffcZrVq1Mj755BPDMAxj165dRqdOnYzc3FwjLi7OGDJkiGEYhrFmzRrj1ltvNU6ePGmcOXPGaN68uZGenm6sXr3aiIyMNFJSUozc3Fyjb9++xieffGL88ccfRufOne3trl692oiLizMMwzD69+9vJCQkGIZRvPM9KSnJuOeeewzDMIycnBzjtttuM86cOWPMnj3bmD17tpGUlGT06tXLvn7UqFFGVlaWff3JkyeN22+/3Thy5IhhGIbx5ptv2vtf586djUWLFhmGYRibNm2yPw9KXkJCgtGsWTMjOTnZMAzDGDJkiLFkyRIjNjbWyMrKMgzDMF577TVj8uTJhmE4njcX/9swDPt76+zZs43777/fvnz48OHGvn37DMMwjO3btxvR0dGGYRhGXFycsXr16gI1XThPrVarERsba5w8edIwDMNYtWqVMXjwYHvbF2r65ZdfjNatWxuGYRhTp041XnzxRcMwDOPIkSNGs2bNjISEBCMhIcHo37+/vY0L5+GZM2cueax5eXlGhw4d7P3izTfftPelUaNGGRs3bjQMwzBOnTpldO3a1V7jxfr37288/fTThs1mM/bs2WNERkYaubm5BV6fmJiYS/b9wYMHG0uWLDEMwzB27NhhDBo0yNi/f7/Rr18/Izs72zAMw5g3b54xderUy/a3J554wvjss88MwzCM9evXG2vWrCnws+rdu7eRm5trZGZmGu3atTN+/fVXY+vWrUafPn2M7OxsIysry+jdu7cxe/bsAscI93dxH8/KyjKaNWtmLF++3DAMwxg7dqyxZMkS+/nwyiuvGJMmTTIM4/y5HRUVZSQlJV32M2bz5s3GsGHDDMMwjNOnTxtjxoxxzUGWcYxYoVQkJCQoOjpafn5+kqSYmBitXbu2wHbdunWTJDVo0ECbN2/W2bNn9dNPP2n8+PH2bfLz83Xs2DFJUvPmza998biu7dixQ3fddZfKlSsnSVqyZIk6deqku+66S5LUrFkzBQUF6cCBA5KkTp06SZKqV6+u+vXrKyQkRJJUsWJFZWRkSJI6d+6sKlWqSJK6d++ub7/9Vk2aNLlqLcU938PDwyVJv//+u/bt26fIyEgFBATY19euXVt5eXl68MEH1bFjRz311FP2vihJP/74oyIiIlSrVi1JUt++fR3+ctqxY0d7O2lpaVc9DhRf/fr1Vb16dUlSo0aNdObMGc2aNUuffPKJDh06pK+//lqNGjUq0nM2a9bM/u/p06fr888/16ZNm/TDDz/o7NmzhXoOs9msefPmaevWrTp48KB27twps/l/l2lfOEcaNWpkP0e++eYbTZ8+XZJUq1Yt3XbbbVdsIyAg4JLHum/fPlWqVMneh/r27atly5bZ2/jtt980d+5cSef7y++//27vlxfr06ePTCaTGjVqpLCwMO3du9fh9Tl79qwOHz58yb6/Y8cO+7G0bt1arVu31tKlS3X48GH17dvX3natWrUu29+ioqI0fvx4de7cWZ07d9Ydd9xRoMa2bdvKx8dHPj4+qlOnjtLT07Vt2zZFR0fL19dXknTPPffY32vgeRo2bGjv48HBwWrbtq2k858pF/9cExISNGXKFElSpUqV1KVLF+3cuVMBAQGX/IwZPHiwpkyZokGDBqljx46Ki4sr5SODxFRAlBKbzVZgWX5+foFlXl7nT0mTyWTfz8fHx2EO+okTJ+xvKBf/cggUh8VisZ9vkpSenl5gG8Mw7Oert7e3ffmF8/XvLl5us9lkNptlMplkGIZ9+blz5wrsV9jz/eLwdcstt2jKlCnq2bOnNmzYoL179+q+++5zeF5/f3+tXbtWO3bs0DfffKN+/frZp2FdaPfvx3txfRdC58WvE66NC6+1dP71/vPPP9WnTx/FxsaqQ4cOqly5spKSki6574XzKy8vz2H5xe+TDzzwgFq3bq3IyEi1bdtWY8aMcdh2+fLlWrFihSSpX79+9nrOnj2rmJgY3XPPPWrVqpUaNmyod999t0DdF58jFovF4Zy/+Lj+3he8vb117NgxPfjggwWO1WKxXPIzRDp/7r7zzjuqWLGiJCklJUWVKlXSo48+qpSUFEmy/5HAYrE47Hfh8YXX51K1Xuj7Xl5eDsf222+/yWq1qnv37va+mJWVpby8vMv2t5iYGLVt21ZffPGFlixZoi+++EIvvviiQ3t///kbhiGz2XzZ44fnufgzRHI8Ly/29/Px4s+hS33GhIaGasOGDdq2bZu+/vpr9erVS+vXr1dgYGAJHwGuhLsColRERkbq448/VnZ2tvLz87V69Wq1atVKFovlkgHrggoVKuiGG26w/6KZmJio3r17X3EfoChat26tzZs3Ky8vT3l5eRoyZIjOnj2rDRs2SJJ2796tlJQUNWjQoNDP+fXXXysjI0O5ublav3692rVrp6CgIKWlpSklJUVWq1WbNm2yb2+xWGS1Wgt9vjdp0kTr1q3TunXr7H/R7NGjhzZu3KikpCS1a9fOYfvExEQ9+uijioyMVFxcnOrVq6eDBw/a1zdt2lQ//vij/vjjD0nSe++9p9atWxfhVcS18tNPP+mGG27QwIED1bRpU3311VeyWq2S/nfeSOf/8v3rr79Kkj799NNLPldaWpoOHTqkJ598Uh07dtS2bdvs+19w//3328+ti29ucujQIZlMJj3++OOKjIx0qONybr/9dn3wwQeSpOPHj2vHjh0ymUwKDg7WoUOHlJ2drezsbH3xxRdXPNYbb7xRZ86c0S+//CJJDrMdIiMj7aNXhw4dUnR0tNLT0/Xmm2/ajyM0NFSStH79ens7aWlpBfp0QECAatWqdcm+37p1a/v+u3bt0qhRo9SmTRtt3rxZJ0+elCS9/PLLmjdv3mX726OPPqqDBw/qgQce0MiRI7Vnz54rvn4Xv44bNmxQbm6u8vLytGHDBv7IUQZERkZq5cqVkqTTp09ry5YtatmypaRLf8Z89NFHmjhxorp06aLx48fL39/fPtsBpYcRK5SKzp07KykpSffdd5/y8/N12223acCAAfrxxx91zz33aNWqVZfdd/r06Zo4caIWLlwoi8Wi1157TT4+PqVYPa5nXbt21Z49exQTEyObzaaYmBh17NhREydO1Lx58+Tt7a34+PginXNVqlTR4MGDdfr0aUVHR9unDw4ZMkT9+vVT5cqV1bJlS50+fVrS+emFEyZM0Msvv1zs871KlSoKDQ1VvXr1CvwF9NZbb9WNN95on07UuHFjdejQwf6LauXKlTV58mQNGzZM+fn5CgsL00svvVTo48W1065dO/3666/q3r27fHx8FBERoX379klyPG8GDx6suLg4ffDBB2rbtq19lPNiFStWVJ8+fXT33XcrICBATZs2VU5OTqGmA4aHh6tx48a666675Ovrq1atWik5OfmSozwXPP7443ruuefUo0cPValSRdWrV5evr6/q16+vbt26KTo6WqGhobr11lslnQ8Qy5cvL3CsPj4+mjlzpp577jkZhqHw8HD7tLjx48drwoQJ6tGjhwzD0JQpUy45DVA6P/rbq1cv2Ww2zZw585Ijzhf639/7/nPPPafx48dr2bJl8vHx0SuvvKLw8HANGzZMAwcOlM1mU7169TR27Fj5+/tfsr+FhIRo0qRJmjFjhry8vAp9G/uOHTvqp59+Uq9evVS+fHkFBwc7jGzh+jR06FBNnDhR0dHRslqteuyxxxQREaH9+/df8jMmNzdXn332me6++255e3vrjjvu4NbtLmAyrvSuCAAokgt3+5s6daqrSwFc6sMPP1RYWJhat26tzMxM9e7dWytXrlRQUFCRnscwDE2bNk1Dhw5VQECAtmzZog8//FCzZ88u9HPExsY63DXRk/zwww/at2+f+vTpI8MwNGLECMXExNj/YIOyhc8Y98aIFQAAKHE33nijJkyYYJ8yOHLkyCKHKun8tUYhISH65z//KW9vb4WEhOiFF14o6XLd1g033KC5c+fqnXfekXR+pJJQBbgnRqwAAAAAwEncvAIAAAAAnESwAgAAAAAnEawAAAAAwEkEKwBAmcPlxQCAkkawAgCUKZ999pni4uJcXQYA4DrD7dYBAGXKkiVLXF0CAOA6xIgVAAAAADiJ77ECAJQZsbGx2rlzp/3xO++8o6CgIM2ZM0eJiYk6c+aMKlWqpDvuuENjxoyRr6+vJCkzM1PTpk3T5s2blZOTo06dOqlp06Z6+eWXtXfvXlcdDgDAjRCsAABlxv79+/Xvf/9bkjRhwgRVqVJF99xzj5o1a6bY2Fj5+Pjoq6++0uLFizV69Gg99thjkqQBAwYoKSlJTz31lKpXr65ly5Zp+/btysvLI1gBACRxjRUAoAy56aabFBAQIElq1qyZvvnmGzVq1Eivvfaaffltt92mbdu2aceOHXrssce0fft27dixQ/Hx8erWrZskqUOHDoqOjtbvv//usmMBALgXghUAoMxq166d2rVrp3Pnzmn//v06fPiw9u3bp9OnT6tixYqSpISEBHl7e6tr1672/cxms7p37674+HgXVQ4AcDcEKwBAmWWz2TRz5ky9++67ysrKUrVq1RQREaFy5crZt0lNTVXFihVlNjve7ykkJKS0ywUAuDGCFQCgzFqwYIGWLFmiSZMmqVu3bqpQoYIk6b777rNvExoaqtTUVNlsNodwderUqVKvFwDgvrjdOgCgTLk4HH333Xe66aabFBMTYw9VJ06c0L59+2Sz2SRJrVu3Vn5+vrZu3WrfzzAMbdmypXQLBwC4NUasAABlSmBgoHbt2qXt27erTp06+uabb7RgwQI1a9ZMhw8f1vz585WXl6fs7GxJUqtWrXT77bdr3LhxOnnypKpXr65Vq1Zp7969MplMLj4aAIC74HbrAIAyJSEhQc8884z++usvvfDCC/rpp5+0adMmnTlzRtWqVdPdd98tk8mk+fPna9u2bQoMDFR6erqmTp2qLVu2KD8/X126dFFgYKDWrl2r77//3tWHBABwAwQrAACuIDk5Wbt371aXLl3sXxgsSSNGjNAff/yhDz74wIXVAQDcBVMBAQC4ArPZrLFjx6pLly667777ZLFY9PXXX2vTpk16+eWXXV0eAMBNMGIFAMBVJCQkaO7cuUpKSlJ+fr7q1aungQMHKjo62tWlAQDcBMEKAAAAAJzE7dYBAAAAwEkEKwAAAABwEsEKAAAAAJxEsAIAAAAAJxGsAAAAAMBJBCsAAAAAcBLBCgAAAACcRLACAAAAACcRrAAAAADASf8PQMJWiyTCF+QAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Zero-shot with GPT 3.5\n",
    "method = \"zero_shot\"\n",
    "model = \"gpt-3.5-turbo-0613\"\n",
    "y_pred[method][model], performance[method][model] = evaluate(\n",
    "    test_df=test_df, model=model, system_content=system_content,\n",
    "    assistant_content=\"\", tags=tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24af6d04-d29e-4adb-a289-4c34c2cc7ec8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 191/191 [06:33<00:00,  2.06s/it] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"precision\": 0.9314722577069027,\n",
      "  \"recall\": 0.9267015706806283,\n",
      "  \"f1\": 0.9271956481845013\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA00AAAE9CAYAAADXvonEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8oElEQVR4nO3de3zP9f//8fv7/d5mmxkz2sgxYZQ5xIycSRSJ8aEyhYpyyuHTVsihlBBl+BSK+gglp0pySDr4Ga1IPpZDhizsg23MTvZ+v35/+Hp/vBtvO9l7m9v1cumSvU7Px+u913Pv3fd8vl5vk2EYhgAAAAAA12V2dQEAAAAAUJQRmgAAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoA4DYUFRWlunXr5ui/yMhIV5erOXPmqG7dutq1a5ck6eTJk6pbt67GjRuXp+OdOXNGqampN93ueu2Eh4erbt26ysrKylPbuakrMjJSdevW1fHjxwu8LQBAzrm5ugAAQOF74IEHVK1aNYdlb7zxhhITEzVjxgyH5X/frigoX768ZsyYoapVq+Z6388++0zTpk3TF198IW9v71vWTkHU1bdvX7Vo0UIVKlS45e0DAG6M0AQAt6GgoCAFBQU5LHvnnXeUmJioHj16uKiqnPP29s5zndHR0TkaZcpvO7l1vboaN26sxo0bF0r7AIAbY3oeAAAAADhBaAIA3NSmTZs0aNAgNW/eXPfcc4+aN2+uoUOHav/+/dm2/fzzz9WrVy81atRIrVu31qxZs7Rq1SqHe5Ju1lafPn3UqFEjtWnTRnPnzpXVanXY5nr3GqWlpemNN95Qly5dFBwcrObNm2vIkCH6+eef7dt06NBBX3zxhSSpY8eOCg8Pl3Tl3qEGDRrou+++U/v27dWgQQONGTPG6b1TBw4cUHh4uIKDg9WyZUu9/PLLSkhIcNimQ4cOatOmTbZ9r74ea9asuWldf7+nyWq16t///rd69Oih4OBgNWnSRAMGDNB3333n0MaaNWtUt25d7dy5UzNmzFC7du107733qkuXLlqyZMlNvgsAgGsxPQ8A4NTSpUv1xhtvqHnz5ho+fLjc3d21f/9+rVu3Trt379bWrVtVvnx5SdLChQv11ltv6Z577tELL7ygixcvatmyZTlua/ny5ZoyZYrq1KmjUaNGKTU1VcuXL1daWtpN9x0zZox27NihJ554QnfddZfOnj2rjz/+WE8++aQ+++wzBQUF6eWXX9bixYu1Z88evfTSS6pdu7Z9/6ysLI0bN079+/dXuXLlFBgY6LS9J598Ui1atFBERIQOHjyoVatWKTo6WmvXrlXZsmVzfM6SnNZ1LZvNpuHDh2vbtm1q3ry5xo4dq0uXLmnNmjV69tlnFRkZqYEDBzrsM378eHl7e2vAgAFyc3PT8uXLNX36dPn4+KhPnz65qhMAbleEJgDADVmtVr377ruqV6+elixZIovFYl/n6+ur999/X7t371aXLl105swZRUVF6Z577tHKlSvl4eEhSerRo4e6d+9+07ZSUlI0c+ZM1a5dW59++qm8vLwkSb169brpfUXnz5/Xtm3b9NhjjykiIsK+PDQ0VJGRkfrtt98UFBSkTp066euvv9aePXvUqVMnValSxb6tzWZT//79NWrUKPuykydP3rDNRx99VJMmTbJ/Xbt2bb322mt6//33NWbMmJue77Wc1XWtzz//XNu2bdOjjz6q6dOny2QySZIGDBigsLAwzZw5Ux07dnR4eEfp0qW1evVq+/ejQ4cO6tixo1avXk1oAoAcYnoeAOCGLBaLvv/+e3344YcOgSk1NVXu7u6SroQdSdq6dasyMzM1aNAg+y/o0pWn7z3yyCM3bWvnzp1KTU1V79697YFJkipVqqRu3bo53dfHx0dlypTRpk2btGrVKv33v/+VdOVBClen++VEy5Ytc7SdJA0bNszh6379+qlMmTLavHlzjo+RW19//bUkaeTIkfbAJF05/yFDhshqtWrTpk0O+zz44IMO348qVarIz89PZ8+evWV1AkBJw0gTAMApDw8P/fzzz9q4caPi4uIUHx+vU6dOyTAMSbL/Py4uTpJUs2bNbMeoVavWTds5ceKEJKlGjRq53t/Dw0PTp0/XSy+9pAkTJkiS6tSpo1atWql79+6qX7/+TduXJH9//xxtV65cuWyPAXd3d1eVKlV0+PDhHB0jL06cOCFvb2/deeed2dZdndL399GxihUrZtvWw8NDNpvt1hQJACUQI00AAKfGjh2rp556Sj///LOqVaum8PBwffDBB3rllVcctsvMzJQkh1GNqzw9PXPcXkZGRrZlV4OZM506ddL333+vqKgo9e3bV5mZmfrggw/Uq1cvffTRRzlq+9rRNGeuHeX5e505OUZePxjX2etwNQT9/fU3m3mrB4D8YqQJAHBDMTEx+vLLL9W1a1fNmTPHISzs3bvXYdurI0xHjx7N9iCDo0eP3rSt6tWr33DbY8eOOd03JSVFBw8eVJUqVdS5c2d17txZkhQbG6sBAwZo/vz5GjBgwE1ryKnk5GRduHBBvr6+9mWZmZn6888/7echXQlh1/tMqLxOjatWrZqOHj2q+Pj4bKNNV0e4KleunKdjAwBujD8/AQBuKCkpSdKVqV/XBqbz58/rs88+k/S/UZPOnTvLzc1Ny5Yt0+XLl+3bJiQk2B+n7cz999+vsmXL6uOPP9aFCxfsy8+dO6f169c73ffgwYN6/PHHtWDBAofltWvXVpkyZeTm9r+/EV4dCcrJ6NWN2Gw2LV++3GHZhx9+qEuXLqlr1672ZXfccYeSkpIcpsxlZGTY7026Vk7qevDBByVJc+fOddju0qVLWrRokSwWizp16pS3kwIA3BAjTQCAG2rSpInKlSunRYsWKT09XdWqVdPJkye1evVqXbx4UZLs/7/zzjv13HPPKSoqSo899pi6detmf2T41dGWG01rkyQvLy9NnjxZY8eOVa9evdS3b18ZhqHly5fbHzrhrM6WLVtq5cqVunDhgkJCQmS1WrVx40bFx8c7PFHv6n1LixcvVuvWrfMUMry8vPTee+/p5MmTatCggfbs2aO1a9fqnnvu0aBBg+zbPfroo4qJidHgwYP1xBNPyGazafXq1dcNRjmpq0ePHvr666+1bt06nTp1Sh07dlRaWppWr16tEydOaNy4capatWquzwcA4ByhCQBwQ+XLl9cHH3yg2bNn69NPP1VmZqYCAgL04IMPauDAgerSpYt++OEHPfvss5Kk4cOHq0KFClq2bJlmzZolPz8/hYWFKSMjQ0uWLLnu/U7Xeuihh+Tn56f58+drwYIF8vT0VPfu3VW9enVNnTr1hvuZTCZFRUXpgw8+0MaNG7V9+3ZJUlBQkGbNmuXwyPP+/fvrl19+0erVqxUdHZ2n0OTr66u3335b06dP1/r161W2bFk9+eSTGjlypMP9W3369FFqaqpWrFihGTNmqEKFCurRo4fatGmjxx9/3OGYOanLYrFowYIF+vDDD7Vu3TrNmjVLXl5eatCggSZOnHjdD9IFAOSfycjP/AQAAP5PamqqrFarypQpk23dxIkT9emnn+qbb7654WcQAQBQVHFPEwCgQBw+fFhNmzbVvHnzHJZfvHhR3377rSpWrHjdR2UDAFDUMT0PAFAg7r33XtWtW1fvvvuuzp8/r3r16ikpKUlr1qzRuXPn9NZbbzm9pwkAgKKK6XkAgAJz/vx5LV68WFu3btXp06fl5eWl4OBgPf3002revLmrywMAIE8ITQAAAADgBPc0AQAAAIAThCYAAAAAcILQBAAAAABOuPzpeVlZWZo/f77WrVunpKQk1a9fX//85z/VqFEjSVJsbKymTZum/fv3q3z58nrqqac0YMCAfLVpGIZsNm7lAgAAAG5nZrMpR092dXlo+te//qVVq1Zp+vTpqlq1qhYtWqSnn35aX331ldzd3TVw4EB16NBBU6ZM0d69ezVlyhSVLl1aYWFheW7TZjN0/vylAjwLAAAAAMVN+fKlZbEUg9C0detWdevWTa1atZIkRUZGatWqVdq7d6/i4uLk7u6uqVOnys3NTbVq1dLx48e1cOHCfIUmAAAAAMgpl9/T5O/vr2+//VYnT56U1WrVJ598Ig8PDwUFBSkmJkYhISFyc/tftgsNDdWxY8d09uxZF1YNAAAA4Hbh8pGm8ePHa9SoUerYsaMsFovMZrOioqJUrVo1nT59WnXq1HHY/o477pAknTp1ShUqVMhzu25uLs+LAAAAAIoBl4emI0eOqEyZMpo/f74CAgK0atUqjRs3TsuWLVN6ero8PDwcti9VqpQkKSMjI89tms0m+fmVzlfdAAAAAG4PLg1Np06d0tixY7V06VI1bdpUktSgQQMdOXJEUVFR8vT0VGZmpsM+V8OSt7d3ntu12QxduJCa98IBAAAAFHu+vl6yWG4+A82loenXX3/V5cuX1aBBA4flDRs21Pfff6/KlSsrISHBYd3VrwMCAvLVdlaWLV/7AwAAALeKzWaT1Zrl6jKKNYvFTWZzwdyS49LQFBgYKEk6ePCggoOD7csPHTqkGjVqqGHDhlq5cqWsVqssFoskKTo6WjVr1pS/v79LagYAAABuFcMwdOHCeaWlpbi6lBLBy8tHvr7lc/RZTM64NDQFBwfrvvvuU0REhCZNmqTAwECtW7dOO3fu1IoVK1SlShUtXrxY48eP19NPP619+/Zp6dKlmjJliivLBgAAAG6Jq4HJx8dPHh6l8v3L/u3KMAxlZmYoJSVRklS2bP4GXEyGYRgFUVheJScn6+2339b27duVnJysOnXqaMyYMQoJCZEk7du3T9OmTdOBAwdUsWJFDRo0SP37989Xm1arjQ+3BQAUKLPZJLOZX25yw2YzZLO59NcQoEix2axKSDgpHx8/+fj4urqcEiEl5YJSUhJ1xx1VrztV78qH2958Cp/LQ5MrEJoAAAXJbDapXDnvHL3x4n+sVpuSklIJTsD/uXw5U+fOnVL58oHy8Cjl6nJKhMzMDJ0/f1r+/pXk7u6RbX1OQ5PLHzkOAEBxZzabZLGYNX/FDsUnJLu6nGLhzjvKathj98tsNhGagL9hSl7BKajXktAEAEABiU9I1rH4RFeXUawwOpd7TGu8PblqCjDX2xWEJgAAUOjKlvGUYbPJ19fL1aUUOzabVYmJafwiextx5RTgvE6jPX36tPbv/1WdOj14iyorXIQmAABQ6Ep7eshkNivuy0VKO3fK1eUUG17+lVSz2zNMa7zNuGoKcH6m0U6bNkmBgZUITQAAAPmVdu6U0s6ccHUZQLFQnKYAl7RnzRGaAAAAABSY4cOf1d69v2jv3l+0Z8/PkqR27ToqOnqHEhPP67XXZuj9999TpUqVNX78ZIf9rl127Fic5s2bo19/3SNvb281adJMw4e/IH//CoV+Ttx9CQAAAKDAvP76TN17b7A6dHhAixZ9JElas+ZTjRo1Tm+9FaV77mlw02OcPftfDRv2tKpUqabFi/+tN998W5cupWjo0EFKS0u71aeQDaEJAAAAQIHx9S0rNzc3lSpVSn5+fpKk0ND71axZcwUF1ZeHR/bPS/q7tWs/U8WKAXrhhXGqXr2GgoLqaerU6Tp//py+/XbrrT6FbJieBwAAAOCWqlKlaq62P3Tod8XF/aEHHmjtsDwzM1PHjsUVZGk5QmgCAAAAcEuVKlXqpttYrVb7v202Q02aNNXYsZHZtvPxKVOgteUE0/MAAAAAFCiTyfkH8bq7u+vSpUv2r202m/7666T967vuqqXjx4/pjjsCVKVKVVWpUlW+vr6aO/ctHT165JbVfSOEJgAAAAAFysvLW6dO/aWEhDPXXX/vvcH66addio7+fzp58k/NmTNTFy+m2Nf37NlbKSkpmjp1gg4fPqTDhw/plVdeUmzsAdWsWauwTsOO6XkA4CJms0lms/O/xMGRzWbwgZ4Ablt33lG22LT36KNhmjZtkp588jF5eXllW9+v3xOKjz+piRMj5eHhrocf7qFOnTrbP9+pcuU7NW/ee3r33Xl6/vnBslgsatCgoebOfdf+cInCRGgCABcwm03y8/OS2WxxdSnFis1mVWJiGsEJwG3FZjNktdo07LH7C71tq9WWp5+5LVu20oYN39xwfenSPnrllVedHqNOnSDNnj0v123fCoQmAHCBK6NMFsV9uUhp5065upxiwcu/kmp2e0Zms4nQBOC2YrMZSkpKdcnsBEb4ryA0AYALpZ07pbQzJ1xdBgCgiCO8uBYPggAAAAAAJwhNAAAAAOAE0/MA5BtPgcs9i4W/WQEAUFwQmgDki9lsUrly3oQAAABQYhGaAOSL2WySxWLW/BU7FJ+Q7Opyio2GdSurb5dGri4DAADkAKEJQIGIT0jWsfhEV5dRbFSu6OvqEgAAQA4xnwYAAAAAnGCkCQAAACjiXPXQJT4f6gpCEwAAAFCEmc0m+fl5yWy2FHrbNptViYlpRTY49e7dXV27dtPgwUNuaTuEJgAAAKAIuzLKZFHcl4uUdu5UobXr5V9JNbs9I7PZVGRDU2EhNAEAAADFQNq5U0o7c8LVZdyWCE0AAAAACkyrVk01evSL2rTpKx05ckhVqlTVs88+r1at2kqS3n//Pe3Z87P8/f21c+f/U9euD2v06Bf122+/6t135yk29oDKlSun++9vo6FDh6l0aR9JUkpKit5+e6Z+/PE7ubm5qX//pwrtnHh6HgAAAIAC9e678/Tggw9p6dLlatGilV5++Z/67bdf7ev37v1F5ctX0JIlH6t37346cuSwXnjheTVv3kIffrhCkyZN08GDsRo9ergM48rUwFdeiVRs7H/05ptzNGfOfO3cuUOnTxfOdEVCEwAAAIAC9dBD3RQW9g9Vq1ZDzz03QkFB9fXZZ584bDN48BDdeWcVVa1aTStWfKSQkFANGDBIVatWU8OGjTR58jQdOLBfe/b8rBMnjmn37miNHv2iGjZsrNq162rSpNfk4eFRKOfD9DwAAAAABapJk6YOXzdoEKzdu6PtX/v5lZePj4/964MHD+rkyRN64IHW2Y51/PgxJScnSZLq1atvX16+vL8qV76zgCu/PkITAAAAgAJlsTjGDKvV5vDI9FKlSjmsNwybOnfuqgEDBmU7VrlyfoqJ2SVJ2Z7i9/d2bhWm5wEAAAAoUL//fsDh6/3796lu3aAbbl+zZi3FxR1VlSpV7f9ZrVbNnTtbCQmnVbt2XUlyuC/q4sWLio//89acwN8w0gQAAACgQH366QpVq1ZDQUH19Pnna3XkyCFFRk684fb9+vXXsGFP66233lRY2D+UknJRb701XRkZGapatbrc3d3Vvn0nzZkzQ+7u7vL399e7787X5cuXC+V8CE0AAABAMeDlX6nYtPfoo7306afLdfToEdWqVVuzZ8/T3XfXvuH2997bQLNnz9Pixf/SoEH95e3tpfvua6Zhw16Qu7u7JGnChMmaN+8dTZr0smw2m3r06KWkpMQ815gbhCYAAACgCLPZDNlsVtXs9owL2rZmu48oJ2rUuEvPPz/quusGDx6iwYOHZFt+333NdN99zW54zFKlPDV2bITGjo3IdT35RWgCAAAAijCbzVBiYprMZpNL2s5LaCppCE0AAABAEUd4cS1CEwAAAIAC8+OPMa4uocDxyHEAAAAAcILQBAAAAABOEJoAAACAIsQwuHepoBTUa0loAgAAAIoAi8UiScrMzHBxJSXH1dfSYsnfoxx4EAQAAABQBJjNFnl5+Sgl5coHtnp4lJLJVPiPGS8JDMNQZmaGUlIS5eXlI7M5f2NFhCYAAACgiPD1LS9J9uCE/PHy8rG/pvlRJELTunXrtHDhQv3555+qVq2ahg8frq5du0qSTp48qVdffVU//fSTvL291bt3b40YMcI+fAkAAACUFCaTSWXL+qtMGT9ZrVmuLqdYs1jc8j3CdJXLQ9P69es1fvx4vfzyy2rdurU2bNigMWPGKDAwUPfee68GDx6sGjVqaOXKlTpx4oTGjx8vs9mskSNHurp0AAAA4JYwm80ymz1cXQb+j0tDk2EYeueddzRgwAA98cQTkqTnnntOMTEx2r17t+Lj4/XXX3/p008/VdmyZVWnTh2dO3dOM2bM0NChQ+XhwYUEAAAA4NZy6dPz4uLiFB8fr+7duzssf//99zVkyBDFxMTonnvuUdmyZe3rQkNDlZKSotjY2MIuFwAAAMBtyKUjTXFxcZKk1NRUDR48WAcOHFCVKlX03HPPqUOHDjp9+rQCAwMd9rnjjjskSadOnVLDhg3z3LabG09bBwqCxUJfQuEqitdcUawJJRvXHFC4XBqaUlJSJEkREREaPny4xo0bp02bNun555/XkiVLlJ6eLl9fX4d9SpUqJUnKyMj78+vNZpP8/ErnvXAAgMv4+nq5ugTA5egHQOFyaWhyd3eXJA0ePFg9e/aUJNWrV08HDhzQkiVL5OnpqczMTId9roYlb2/vPLdrsxm6cCE1z/sD+B+LxcybNwrVhQtpslptri7DAf0Aha0o9gOgOPL19crRyK1LQ1NAQIAkqU6dOg7L7777bm3fvl0hISE6dOiQw7qEhASHffMqK4sfNABQHFmtNn6G47ZHPwAKl0snxN5zzz0qXbq0fv31V4flhw4dUrVq1dSsWTMdOHDAPo1PkqKjo1W6dGkFBQUVdrkAAAAAbkMuDU2enp56+umnNX/+fH355Zc6ceKE/vWvf2nHjh0aOHCgOnXqpIoVK+qFF17Q77//rq1bt2r27NkaNGgQjxsHAAAAUChc/uG2zz//vLy8vDRnzhydOXNGtWrVUlRUlJo3by5JWrx4saZMmaJ//OMfKlu2rB5//HE9//zzLq4aAAAAwO3C5aFJkgYOHKiBAwded1316tX1wQcfFHJFAAAAAHAFD/kHAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtAEAAAAAE4QmgAAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACccMvtDi+99FKOtzWZTHr99ddz2wQAAAAAFBm5Dk2nT5/WgQMHlJycrDvvvFMBAQFKSkrS8ePHZRiGAgMD7duaTKYCLRYAAAAACluuQ9NDDz2kw4cPa/ny5WrSpIl9+dGjR/Xcc8/p8ccf15NPPlmgRQIAAACAq+T6nqZ3331X48aNcwhMknTXXXfphRde0Pvvv19gxQEAAACAq+U6NJ0/f15ly5a9/sHMZl28eDHfRQEAAABAUZHr0NSwYUPNmzdPiYmJDssTEhIUFRWlVq1aFVhxAAAAAOBqub6nKTIyUv3791eHDh3UuHFj+fn56dy5c9qzZ4/8/f318ssv34o6AQAAAMAlcj3SFBQUpA0bNqhfv35KSUnR/v37lZ6erkGDBmnNmjWqVKnSragTAAAAAFwi1yNNkhQQEKCIiIiCrgUAAAAAipxcjzRJUmZmppYvX67hw4erb9+++uOPP7RixQrt27cvX8XExcWpcePGWrNmjX1ZbGys+vfvr0aNGqlDhw766KOP8tUGAAAAAORGnp6eFxYWpmnTpun48ePat2+f0tPTtX37doWHh2vPnj15KuTy5csaN26cUlNT7csSExM1cOBAVatWTatXr9awYcM0a9YsrV69Ok9tAAAAAEBu5To0zZgxQ5cuXdJXX32ltWvXyjAMSdLcuXPVoEEDzZ07N0+FREVFycfHx2HZp59+Knd3d02dOlW1atVSWFiYnnrqKS1cuDBPbQAAAABAbuU6NH377bcaNWqUqlevLpPJZF9eqlQpDRo0SP/5z39yXcRPP/2kTz75RNOnT3dYHhMTo5CQELm5/e/Wq9DQUB07dkxnz57NdTsAAAAAkFu5fhBERkaGypUrd911FotFly9fztXxLly4oBdffFETJkzI9uS906dPq06dOg7L7rjjDknSqVOnVKFChVy1dS03tzzdzgXgbywW+hIKV1G85opiTSjZuOaAwpXr0NSgQQMtX75cbdu2zbbuiy++0L333pur402ePFmNGzdW9+7ds61LT0+Xh4eHw7JSpUpJuhLe8spsNsnPr3Se9wcAuI6vr5erSwBcjn4AFK5ch6ZRo0bpqaeeUo8ePdS2bVuZTCZ9+eWXioqK0o8//qjFixfn+Fjr1q1TTEyMvvjii+uu9/T0VGZmpsOyq2HJ29s7t6Xb2WyGLlxIvfmGAG7KYjHz5o1CdeFCmqxWm6vLcEA/QGEriv0AKI58fb1yNHKb69DUtGlTLVmyRG+99ZYWL14swzC0dOlS1a9fX++9955CQ0NzfKzVq1fr3LlzateuncPySZMm6auvvlJgYKASEhIc1l39OiAgILelO8jK4gcNABRHVquNn+G47dEPgMKV69C0c+dONW7cWCtXrlR6erqSk5Pl4+Oj0qVzP91t1qxZSk9Pd1jWuXNnjRw5Uo888ojWr1+vlStXymq1ymKxSJKio6NVs2ZN+fv757o9AAAAAMitXN9FOGLECG3evFnSlelzAQEBeQpM0pXRourVqzv8J0n+/v4KCAhQWFiYUlJSNH78eB05ckRr1qzR0qVLNWTIkDy1BwAAAAC5levQ5OvrK09Pz1tRSzb+/v5avHix4uLi1LNnT82bN08vvviievbsWSjtAwAAAECup+cNGTJEr732muLi4hQUFHTdBzI0a9YszwUdPHjQ4evg4GB98skneT4eAAAAAORHjkJTRkaG/VHfkyZNkiTNmTNHkhw+4NYwDJlMJsXGxhZ0nQAAAADgEjkKTR06dNC8efPUuHFjNWvWTH369FFgYOCtrg0AAAAAXC5HoenixYv2R33HxMTon//8p4KDg29pYQAAAABQFOQoNDVo0EBjx47Vm2++KcMwNGzYMHl4eFx3W5PJpK1btxZokQAAAADgKjkKTbNnz9bSpUuVlJSktWvXqn79+ipfvvytrg0AAAAAXC5HoSkgIEARERGSpF27dmn06NEKCgq6pYUBAAAAQFGQ60eOb9u27VbUAQAAAABFUq4/3BYAAAAAbieEJgAAAABwgtAEAAAAAE4QmgAAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJxwc3UBuD2ZzSaZzSZXl1Gs2GyGbDbD1WUAAADcdghNKHRms0l+fl4ymy2uLqVYsdmsSkxMIzgBAAAUMkITCt2VUSaL4r5cpLRzp1xdTrHg5V9JNbs9I7PZRGgCAAAoZIQmuEzauVNKO3PC1WUAAAAATvEgCAAAAABwgtAEAAAAAE4QmgAAAADACUITAAAAADhBaAIAAAAAJ3h6Xj7xIa25Z7GQ1QEAAFB8EJrywWw2qVw5b0IAAAAAUIIRmvLBbDbJYjFr/oodik9IdnU5xUbDupXVt0sjV5cBAAAA5AihqQDEJyTrWHyiq8soNipX9HV1CQAAAECOMa8MAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4ITLQ1NSUpJeeeUVtWnTRk2aNNFjjz2mmJgY+/qdO3eqV69eatiwobp06aINGza4sFoAAAAAtxuXh6YxY8Zoz549mj17tlavXq169epp8ODBOnr0qP744w8NGTJErVu31po1a9SnTx+9+OKL2rlzp6vLBgAAAHCbcHNl48ePH9eOHTu0fPly3XfffZKkiRMn6ocfftAXX3yhc+fOqW7duho9erQkqVatWjpw4IAWL16sFi1auLJ0AAAAALcJl440+fn5aeHChWrQoIF9mclkkslk0oULFxQTE5MtHIWGhurnn3+WYRiFXS4AAACA25BLQ5Ovr6/atm0rDw8P+7JNmzbp+PHjat26tU6fPq3AwECHfe644w6lpaUpMTGxsMsFAAAAcBty6fS8v/vll1/00ksvqXPnzmrXrp3S09MdApUk+9eZmZn5asvNLf950WJx+S1huM0UxWuuKNaEkq0oXnNFsSaUbFxzQOEqMqFp69atGjdunJo0aaJZs2ZJkkqVKpUtHF392svLK89tmc0m+fmVznuxgIv4+ub9ugdKCvoBQD8ACluRCE3Lli3TtGnT1KVLF7355pv20aRKlSopISHBYduEhAR5e3urTJkyeW7PZjN04UJqvmqWrvyVhx9aKEwXLqTJarW5ugwH9AMUNvoBUDT7AVAc+fp65Wjk1uWhafny5Xr11VcVHh6u8ePHy2Qy2dc1bdpUu3fvdtg+OjpaTZo0kdmcv2HprCx+0KD4sVptXLu47dEPAPoBUNhcGpri4uL0+uuv64EHHtCQIUN09uxZ+zpPT0+Fh4erZ8+emjVrlnr27KnvvvtOX3/9tRYvXuzCqgEAAADcTlwamjZt2qTLly9ry5Yt2rJli8O6nj17avr06VqwYIFmzpypDz/8UFWqVNHMmTP5jCYAAAAAhcaloWno0KEaOnSo023atGmjNm3aFFJFAAAAAOCI51UCAAAAgBOEJgAAAABwgtAEAAAAAE4QmgAAAADACZd/ThMAAABKBrPZJLPZdPMNYWezGbLZDFeXgZsgNAEAACDfzGaTypXzlsXCRKbcsFptSkpKJTgVcYQmAAAA5JvZbJLFYtb8FTsUn5Ds6nKKhTvvKKthj90vs9lEaCriCE0AAAAoMPEJyToWn+jqMoACxfgpAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABO8DlNAAAAgAtZLIxj5JbNZhTqBwITmgAAAAAXKFvGU4bNJl9fL1eXUuzYbFYlJqYVWnAiNAEAAAAuUNrTQyazWXFfLlLauVOuLqfY8PKvpJrdnpHZbCI0AQAAALeDtHOnlHbmhKvLgBNMoAQAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtAEAAAAAE4QmgAAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOBEsQhNNptNc+fOVevWrdWoUSM988wz+vPPP11dFgAAAIDbQLEITQsWLNDy5cv16quvauXKlbLZbHr66aeVmZnp6tIAAAAAlHBFPjRlZmbqgw8+0MiRI9WuXTsFBQVpzpw5On36tDZv3uzq8gAAAACUcEU+NP3++++6dOmSWrRoYV/m6+ur+vXr66effnJhZQAAAABuBybDMAxXF+HM5s2bNWLECP3666/y9PS0Lx81apTS09P13nvv5fqYhmHIZsv/aZtMktlsVnJKuqxWW76Pd7vwcLfIx7uULl+6IMNmdXU5xYLJbJF7aV/ZbDYVtR5LP8gb+kHu0Q9KFvpA3tAPShb6Qd4UZD8wm00ymUw33c4tf83cemlpaZIkDw8Ph+WlSpVScnJyno5pMplksdz8xcmpsj6eN98I2biX9nV1CcWO2Vx0B4fpB3lDP8g9+kHJQh/IG/pByUI/yJvC7AdFt8f9n6ujS39/6ENGRoa8vLxcURIAAACA20iRD02VKlWSJCUkJDgsT0hIUEBAgCtKAgAAAHAbKfKhKSgoSD4+Ptq1a5d92YULF3TgwAE1a9bMhZUBAAAAuB0U+XuaPDw81L9/f82aNUvly5fXnXfeqZkzZyowMFCdO3d2dXkAAAAASrgiH5okaeTIkcrKytKECROUnp6uZs2a6f3335e7u7urSwMAAABQwhX5R44DAAAAgCsV+XuaAAAAAMCVCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtAEAAAAAE4QmgAAAADACUITbpnw8HD7v+vWrevCSgDXmjt3rmJiYgr8uCtWrNCKFSvyvB5FV0FeM5GRkVqzZk225WvWrFFkZGSBtIHr69Gjh6tLQBHD70PFl5urC0DJtXv3bleXABQJP/30k5o3b17gx33sscfytR5F1626ZlC41q9f7+oSABQQQhMKxLvvvqvPP/9cFotF999/v9LS0iRJvXr1sv+Fc8qUKdqzZ48yMjL05ptvKjg4WCdOnNDkyZOVmJgoDw8PRUREqEmTJoqMjFRiYqJOnDihUaNGqUuXLq48PRRzs2fP1qZNm2SxWNSjRw917txZr7zyipKSkuTt7a3x48crODhYkZGR8vT01N69e5WUlKTRo0dr69atio2NVfv27TV+/HitWbNGmzZtUkpKihISEtS2bVuNHz9e8fHxGjBggLZt2ybpyl/xd+/erWbNmmn//v2aMGGC5s6dq9KlS+f6mp8+fbrKly+vZ599VpIUERGhkJAQ/fXXX5KkYcOG2fuXxWJRhw4dNGLECEVFRUmSRowYoW+//VZvv/22bDabqlatqqlTp6pChQrq0KGDevTooR07digpKUkTJ05U69atC/k7VPLt2rVL//rXv1SmTBn98ccfCgwM1OzZs/Xll19q3bp1Sk9Pl3TlWv3tt98crpnXX39dw4cPt4eounXr6uDBg4qKitLevXt1+vRp9enTR/Xr19fs2bOVkZGh5ORkjR07Vg8//HCO6tu4caOWLFmi9PR0paena+rUqQoNDVV4eLgaNmyomJgYJSQkaMSIEerZs6dSUlL08ssv6/Dhw6pYsaJMJpOef/55SdK8efP073//W5IcrsFly5ZlO9c6deooJiZGU6dOldlsVtOmTfXdd99py5YtOn/+vF555RX7dT58+HB16NAhW+3h4eGqWbOm9u/fr7S0NEVGRqpt27bZXp+2bdtet9+fOnVKL730ks6ePSsPDw9NnjxZwcHBWr9+vT788ENZrVbdfffdmjJliry9va/b1/bu3avXXntNhmGoVKlSeu2113TXXXc5fK/OnDmjEydOKD4+3l6LJL3zzjvasGGDypQpo1q1aqlq1aoaMWJEXi81uNCuXbs0f/58ubm56dixY2rdurUCAgK0detW2Ww2LVy40L5tWlqaJkyYoIMHD8pkMmnw4MF69NFHb/gek5GRoRdffFEnTpyQyWRS37591a9fPxee7W3IAPJp+/btRlhYmJGammpcvnzZGDp0qLFs2TKjTp069m3q1KljbNiwwTAMw/jwww+NESNGGIZhGP369TP27dtnGIZhHD9+3Gjfvr1x+fJlIyIiwhg7dmzhnwxKnE2bNhl9+/Y10tPTjfT0dKN3795Gs2bNjK+++sowDMPYs2eP0a5dOyMjI8OIiIgwhg4dahiGYaxZs8a47777jLNnzxoXL140GjdubCQnJxurV682QkNDjYSEBCMjI8Po27ev8dVXXxl//vmn0b59e3u7q1evNiIiIgzDMIz+/fsb0dHRhmHk7ZqPjY01HnnkEcMwDCM9Pd1o2bKlcfHiRWPu3LnG3LlzjdjYWKNnz5729WPGjDFSU1Pt68+ePWvcf//9xokTJwzDMIxFixbZ+2D79u2N999/3zAMw9i8ebP9OChY0dHRRqNGjYz4+HjDMAxj6NChxtKlS43w8HAjNTXVMAzDeOedd4ypU6cahuF4zVz7b8Mw7D9b586dazz22GP25SNGjDAOHTpkGIZh7Ny50+jWrZthGIYRERFhrF69OltNV69Rq9VqhIeHG2fPnjUMwzA+++wzY8iQIfa2r9b0n//8xwgJCTEMwzCmT59uvPbaa4ZhGMaJEyeMRo0aGdHR0UZ0dLTRv39/extXr8GLFy9e91wzMzONNm3a2PvEokWL7P1ozJgxxqZNmwzDMIxz584ZnTp1std4rf79+xsvvviiYbPZjAMHDhihoaFGRkZGttcnLCzsuv1+yJAhxtKlSw3DMIxdu3YZgwcPNo4cOWL069fPSEtLMwzDMBYsWGBMnz79hn3t+eefN7755hvDMAxjw4YNxpo1a7J9r3r16mVkZGQYKSkpRqtWrYzff//d2LZtm9GnTx8jLS3NSE1NNXr16mXMnTs32zmieLi2n6emphqNGjUyVqxYYRiGYURGRhpLly61XxNvvvmmMWXKFMMwrlzfHTp0MGJjY2/4HrNlyxZj+PDhhmEYxvnz541x48a55iRvY4w0Id+io6PVrVs3eXl5SZLCwsK0bt26bNt17txZklSnTh1t2bJFly5d0m+//aYJEybYt8nKytKpU6ckSY0bN771xaPE27Vrl7p27apSpUpJkpYuXap27dqpa9eukqRGjRqpbNmyOnr0qCSpXbt2kqTKlSurdu3a8vf3lySVK1dOFy5ckCS1b99eFStWlCQ99NBD+umnn9SgQYOb1pLXaz4oKEiS9Mcff+jQoUMKDQ2Vj4+PfX21atWUmZmpJ554Qm3bttXo0aPt/VGS9u3bp+DgYFWtWlWS1LdvX4e/eLZt29beTlJS0k3PA3lTu3ZtVa5cWZJUr149Xbx4UXPmzNFXX32lY8eO6YcfflC9evVydcxGjRrZ/z1z5kx9++232rx5s3799VddunQpR8cwm81asGCBtm3bpri4OO3evVtm8/9ueb56fdSrV89+ffz444+aOXOmJKlq1apq2bKl0zZ8fHyue66HDh1S+fLl7f2nb9++Wr58ub2Nw4cPa/78+ZKu9JU//vjD3iev1adPH5lMJtWrV0+BgYE6ePCgw+tz6dIlHT9+/Lr9fteuXfZzCQkJUUhIiJYtW6bjx4+rb9++9rarVq16w77WoUMHTZgwQe3bt1f79u314IMPZquxRYsW8vDwkIeHh6pXr67k5GTt2LFD3bp1k6enpyTpkUcesf+cQfFUt25dez/38/NTixYtJF15T7n2exsdHa1p06ZJksqXL6+OHTtq9+7d8vHxue57zJAhQzRt2jQNHjxYbdu2VURERCGfGQhNyDebzZZtWVZWVrZlbm5XLjeTyWTfz8PDw2HO95kzZ+w/KK79pQ/IK4vFYr/mJCk5OTnbNoZh2K9Zd3d3+/Kr1+zfXbvcZrPJbDbLZDLJMAz78suXL2fbL6fX/LXB6t5779W0adPUo0cPbdy4UQcPHlTv3r0djuvt7a1169Zp165d+vHHH9WvXz/79Kir7f79fK+t72qgvPZ1QsG7+jpLV17rv/76S3369FF4eLjatGmjChUqKDY29rr7Xr22MjMzHZZf+3Py8ccfV0hIiEJDQ9WiRQuNGzfOYdsVK1Zo5cqVkqR+/frZ67l06ZLCwsL0yCOPqFmzZqpbt64+/vjjbHVfe31YLBaH6/3a8/p7P3B3d9epU6f0xBNPZDtXi8Vy3fcQ6cp1+9FHH6lcuXKSpISEBJUvX17PPPOMEhISJMke/i0Wi8N+V7+++vpcr9ar/d7Nzc3h3A4fPiyr1aqHHnrI3g9TU1OVmZl5w74WFhamFi1aaPv27Vq6dKm2b9+u1157zaG9v3//DcOQ2Wy+4fmjeLr2PURyvDav9fdr8tr3oeu9xwQEBGjjxo3asWOHfvjhB/Xs2VMbNmyQr69vAZ8BboSn5yHfQkND9eWXXyotLU1ZWVlavXq1mjVrJovFct3wdFWZMmVUo0YN+y+QMTEx6tWrl9N9gNwKCQnRli1blJmZqczMTA0dOlSXLl3Sxo0bJUl79+5VQkKC6tSpk+Nj/vDDD7pw4YIyMjK0YcMGtWrVSmXLllVSUpISEhJktVq1efNm+/YWi0VWqzXH13yDBg20fv16rV+/3v6XyO7du2vTpk2KjY1Vq1atHLaPiYnRM888o9DQUEVERKhWrVqKi4uzr2/YsKH27dunP//8U5L0ySefKCQkJBevIm6F3377TTVq1NDAgQPVsGFDff/997JarZL+d81IV/5a/fvvv0uSvv766+seKykpSceOHdMLL7ygtm3baseOHfb9r3rsscfs19W1Dwk5duyYTCaTnnvuOYWGhjrUcSP333+/1q5dK0k6ffq0du3aJZPJJD8/Px07dkxpaWlKS0vT9u3bnZ7rXXfdpYsXL+o///mPJDnMUggNDbWPOh07dkzdunVTcnKyFi1aZD+PgIAASdKGDRvs7SQlJWXrzz4+Pqpatep1+31ISIh9/z179mjMmDFq3ry5tmzZorNnz0qS3njjDS1YsOCGfe2ZZ55RXFycHn/8cY0aNUoHDhxw+vpd+zpu3LhRGRkZyszM1MaNG/njxW0iNDRUq1atkiSdP39eW7duVdOmTSVd/z3miy++0OTJk9WxY0dNmDBB3t7e9lkKKByMNCHf2rdvr9jYWPXu3VtZWVlq2bKlBgwYoH379umRRx7RZ599dsN9Z86cqcmTJ2vx4sWyWCx655135OHhUYjVo6Tr1KmTDhw4oLCwMNlsNoWFhalt27aaPHmyFixYIHd3d0VFReXquqtYsaKGDBmi8+fPq1u3bvYpfUOHDlW/fv1UoUIFNW3aVOfPn5d0ZcrfpEmT9MYbb+T5mq9YsaICAgJUq1atbH+5vO+++3TXXXfZp/nUr19fbdq0sf8iWqFCBU2dOlXDhw9XVlaWAgMD9frrr+f4fHFrtGrVSr///rseeugheXh4KDg4WIcOHZLkeM0MGTJEERERWrt2rVq0aGEfmbxWuXLl1KdPHz388MPy8fFRw4YNlZ6enqMpekFBQapfv766du0qT09PNWvWTPHx8dcdnbnqueee08SJE9W9e3dVrFhRlStXlqenp2rXrq3OnTurW7duCggI0H333SfpSjhYsWJFtnP18PDQ7NmzNXHiRBmGoaCgIPtUtQkTJmjSpEnq3r27DMPQtGnTrjs1T7oyYtuzZ0/ZbDbNnj37uqPEV/ve3/v9xIkTNWHCBC1fvlweHh568803FRQUpOHDh2vgwIGy2WyqVauWIiMj5e3tfd2+5u/vrylTpmjWrFlyc3PL8aPc27Ztq99++009e/ZU6dKl5efn5zAihZJr2LBhmjx5srp16yar1apnn31WwcHBOnLkyHXfYzIyMvTNN9/o4Ycflru7ux588EEeX17ITIazn4oAAAdXn4o3ffp0V5cCuMznn3+uwMBAhYSEKCUlRb169dKqVatUtmzZXB3HMAzNmDFDw4YNk4+Pj7Zu3arPP/9cc+fOzfExwsPDHZ4uWJz8+uuvOnTokPr06SPDMDRy5EiFhYXZ/xCD2w/vMUUXI00AACBX7rrrLk2aNMk+jW/UqFG5DkzSlXt7/P399Y9//EPu7u7y9/fXq6++WtDlFlk1atTQ/Pnz9dFHH0m6MsJIYAKKJkaaAAAAAMAJHgQBAAAAAE4QmgAAAADACUITAAAAADhBaAIAlDjcrgsAKEiEJgBAifLNN98oIiLC1WUAAEoQHjkOAChRli5d6uoSAAAlDCNNAAAAAOAEn9MEACgxwsPDtXv3bvvXH330kcqWLat58+YpJiZGFy9eVPny5fXggw9q3Lhx8vT0lCSlpKRoxowZ2rJli9LT09WuXTs1bNhQb7zxhg4ePOiq0wEAFBGEJgBAiXHkyBH985//lCRNmjRJFStW1COPPKJGjRopPDxcHh4e+v7777VkyRKNHTtWzz77rCRpwIABio2N1ejRo1W5cmUtX75cO3fuVGZmJqEJAMA9TQCAkuPuu++Wj4+PJKlRo0b68ccfVa9ePb3zzjv25S1bttSOHTu0a9cuPfvss9q5c6d27dqlqKgode7cWZLUpk0bdevWTX/88YfLzgUAUHQQmgAAJVarVq3UqlUrXb58WUeOHNHx48d16NAhnT9/XuXKlZMkRUdHy93dXZ06dbLvZzab9dBDDykqKspFlQMAihJCEwCgxLLZbJo9e7Y+/vhjpaamqlKlSgoODlapUqXs2yQmJqpcuXIymx2fjeTv71/Y5QIAiihCEwCgxFq4cKGWLl2qKVOmqHPnzipTpowkqXfv3vZtAgIClJiYKJvN5hCczp07V+j1AgCKJh45DgAoUa4NPj///LPuvvtuhYWF2QPTmTNndOjQIdlsNklSSEiIsrKytG3bNvt+hmFo69athVs4AKDIYqQJAFCi+Pr6as+ePdq5c6eqV6+uH3/8UQsXLlSjRo10/Phxvffee8rMzFRaWpokqVmzZrr//vs1fvx4nT17VpUrV9Znn32mgwcPymQyufhsAABFAY8cBwCUKNHR0XrppZf03//+V6+++qp+++03bd68WRcvXlSlSpX08MMPy2Qy6b333tOOHTvk6+ur5ORkTZ8+XVu3blVWVpY6duwoX19frVu3Tr/88ourTwkA4GKEJgDAbS0+Pl579+5Vx44d7R92K0kjR47Un3/+qbVr17qwOgBAUcD0PADAbc1sNisyMlIdO3ZU7969ZbFY9MMPP2jz5s164403XF0eAKAIYKQJAHDbi46O1vz58xUbG6usrCzVqlVLAwcOVLdu3VxdGgCgCCA0AQAAAIATPHIcAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACc+P+x2etO0VkUGgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Zero-shot with GPT 4\n",
    "method = \"zero_shot\"\n",
    "model = \"gpt-4-0613\"\n",
    "y_pred[method][model], performance[method][model] = evaluate(\n",
    "    test_df=test_df, model=model, system_content=system_content,\n",
    "    assistant_content=\"\", tags=tags)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "483f6d46-7a9e-4bce-a34f-96c1cf2df29a",
   "metadata": {},
   "source": [
    "### Few-shot learning"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8d6159b2",
   "metadata": {},
   "source": [
    "Now, we'll be adding a `assistant_context` with a few samples from our training data for each class. The intuition here is that we're giving the model a few examples (few-shot learning) of what each class looks like so that it can learn to generalize better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e22ed1e1-b34d-43d1-ae8b-32b1fd5be53d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'title': 'Comparison between YOLO and RCNN on real world videos',\n",
       "  'description': 'Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.',\n",
       "  'tag': 'computer-vision'},\n",
       " {'title': 'Show, Infer & Tell: Contextual Inference for Creative Captioning',\n",
       "  'description': 'The beauty of the work lies in the way it architects the fundamental idea that humans look at the overall image and then individual pieces of it.\\r\\n',\n",
       "  'tag': 'computer-vision'},\n",
       " {'title': 'Awesome Graph Classification',\n",
       "  'description': 'A collection of important graph embedding, classification and representation learning papers with implementations.',\n",
       "  'tag': 'other'},\n",
       " {'title': 'Awesome Monte Carlo Tree Search',\n",
       "  'description': 'A curated list of Monte Carlo tree search papers with implementations. ',\n",
       "  'tag': 'other'},\n",
       " {'title': 'Rethinking Batch Normalization in Transformers',\n",
       "  'description': 'We found that NLP batch statistics exhibit large variance throughout training, which leads to poor BN performance.',\n",
       "  'tag': 'natural-language-processing'},\n",
       " {'title': 'ELECTRA: Pre-training Text Encoders as Discriminators',\n",
       "  'description': 'PyTorch implementation of the electra model from the paper: ELECTRA - Pre-training Text Encoders as Discriminators Rather Than Generators',\n",
       "  'tag': 'natural-language-processing'},\n",
       " {'title': 'Pytest Board',\n",
       "  'description': 'Continuous pytest runner with awesome visualization.',\n",
       "  'tag': 'mlops'},\n",
       " {'title': 'Debugging Neural Networks with PyTorch and W&B',\n",
       "  'description': 'A closer look at debugging common issues when training neural networks.',\n",
       "  'tag': 'mlops'}]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create additional context with few samples from each class\n",
    "num_samples = 2\n",
    "additional_context = []\n",
    "cols_to_keep = [\"title\", \"description\", \"tag\"]\n",
    "for tag in tags:\n",
    "    samples = train_df[cols_to_keep][train_df.tag == tag][:num_samples].to_dict(orient=\"records\")\n",
    "    additional_context.extend(samples)\n",
    "additional_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "294548a5-9edf-4dea-ab8d-dc7464246810",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here are some examples with the correct labels: [{'title': 'Comparison between YOLO and RCNN on real world videos', 'description': 'Bringing theory to experiment is cool. We can easily train models in colab and find the results in minutes.', 'tag': 'computer-vision'}, {'title': 'Show, Infer & Tell: Contextual Inference for Creative Captioning', 'description': 'The beauty of the work lies in the way it architects the fundamental idea that humans look at the overall image and then individual pieces of it.\\r\\n', 'tag': 'computer-vision'}, {'title': 'Awesome Graph Classification', 'description': 'A collection of important graph embedding, classification and representation learning papers with implementations.', 'tag': 'other'}, {'title': 'Awesome Monte Carlo Tree Search', 'description': 'A curated list of Monte Carlo tree search papers with implementations. ', 'tag': 'other'}, {'title': 'Rethinking Batch Normalization in Transformers', 'description': 'We found that NLP batch statistics exhibit large variance throughout training, which leads to poor BN performance.', 'tag': 'natural-language-processing'}, {'title': 'ELECTRA: Pre-training Text Encoders as Discriminators', 'description': 'PyTorch implementation of the electra model from the paper: ELECTRA - Pre-training Text Encoders as Discriminators Rather Than Generators', 'tag': 'natural-language-processing'}, {'title': 'Pytest Board', 'description': 'Continuous pytest runner with awesome visualization.', 'tag': 'mlops'}, {'title': 'Debugging Neural Networks with PyTorch and W&B', 'description': 'A closer look at debugging common issues when training neural networks.', 'tag': 'mlops'}]\n"
     ]
    }
   ],
   "source": [
    "# Add additional context\n",
    "assistant_content = f\"\"\"Here are some examples with the correct labels: {additional_context}\"\"\"\n",
    "print (assistant_content)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a087e14f",
   "metadata": {},
   "source": [
    "> We could increase the number of samples by increasing the context length. We could also retrieve better few-shot samples by extracting examples from the training data that are similar to the current sample (ex. similar unique vocabulary)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29bca273-3ea8-4ce0-9fa9-fe19062b7c5b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 191/191 [01:16<00:00,  2.49it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"precision\": 0.8435247936255214,\n",
      "  \"recall\": 0.8586387434554974,\n",
      "  \"f1\": 0.8447984162323493\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA00AAAE9CAYAAADXvonEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9EklEQVR4nO3df3zN9f//8fs5Z5ttZrMNQ6gIo8yP/BjJ75CGGG/EFCoKyY+3rZAfpYQ3ZegHRb2F8ruS/Eg/fYxWFG+L5FctLOyH2S875/X9w3cnJxz7ZWfmdr1cuuS8fj0fr7PX85xzP8/X63VMhmEYAgAAAABcldnVBQAAAABAcUZoAgAAAAAnCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtAEAAAAAE4QmgAAAADACUITAAAAADhBaAKAW1B0dLRq166dq/+ioqJcXa7mzp2r2rVra9euXZKkP/74Q7Vr19a4cePytb3Tp08rLS3tustdrZ2IiAjVrl1b2dnZ+Wo7L3VFRUWpdu3aOn78eKG3BQDIPTdXFwAAKHoPPPCAqlWr5jDtlVdeUWJiombOnOkw/Z/LFQcBAQGaOXOmqlatmud1V69erenTp+uTTz6Rt7f3DWunMOrq06ePmjdvrnLlyt3w9gEA10ZoAoBbUHBwsIKDgx2mvf7660pMTFT37t1dVFXueXt757vOmJiYXI0yFbSdvLpaXQ0bNlTDhg2LpH0AwLVxeh4AAAAAOEFoAgBc1+bNmzV48GA1a9ZMd999t5o1a6Zhw4Zp//79Vyz78ccfq2fPnmrQoIHuv/9+zZ49W6tWrXK4Jul6bfXu3VsNGjRQq1atNG/ePFmtVodlrnatUXp6ul555RV17txZISEhatasmYYOHaoffvjBvky7du30ySefSJLat2+viIgISZeuHapXr56+/vprtW3bVvXq1dOYMWOcXjt14MABRUREKCQkRC1atNDzzz+vhIQEh2XatWunVq1aXbFuzvOxdu3a69b1z2uarFar/vvf/6p79+4KCQlRo0aNNHDgQH399dcObaxdu1a1a9fWzp07NXPmTLVp00b33HOPOnfurCVLllznrwAAuByn5wEAnFq6dKleeeUVNWvWTCNGjJC7u7v279+v9evXa/fu3dq2bZsCAgIkSW+//bb+85//6O6779azzz6r8+fPa9myZblua/ny5Zo6dapq1aqlUaNGKS0tTcuXL1d6evp11x0zZox27Nih/v37q3r16jpz5ow++OADPfroo1q9erWCg4P1/PPPa/HixdqzZ4+ee+451axZ075+dna2xo0bpwEDBqhs2bKqWLGi0/YeffRRNW/eXJGRkTp48KBWrVqlmJgYrVu3Tn5+frneZ0lO67qczWbTiBEjtH37djVr1kxjx47VhQsXtHbtWj355JOKiorSoEGDHNaZMGGCvL29NXDgQLm5uWn58uWaMWOGfHx81Lt37zzVCQC3KkITAOCarFar3nzzTdWpU0dLliyRxWKxz/P19dU777yj3bt3q3Pnzjp9+rSio6N19913a+XKlfLw8JAkde/eXV27dr1uW6mpqZo1a5Zq1qypjz76SF5eXpKknj17Xve6onPnzmn79u3q16+fIiMj7dNDQ0MVFRWlffv2KTg4WB06dNDnn3+uPXv2qEOHDqpSpYp9WZvNpgEDBmjUqFH2aX/88cc123z44Yc1efJk++OaNWvqpZde0jvvvKMxY8Zcd38v56yuy3388cfavn27Hn74Yc2YMUMmk0mSNHDgQIWHh2vWrFlq3769w807SpcurTVr1tj/Hu3atVP79u21Zs0aQhMA5BKn5wEArsliseibb77Re++95xCY0tLS5O7uLulS2JGkbdu2KSsrS4MHD7Z/QJcu3X2vW7du121r586dSktLU69eveyBSZIqVaqksLAwp+v6+PioTJky2rx5s1atWqW//vpL0qUbKeSc7pcbLVq0yNVykjR8+HCHx3379lWZMmW0ZcuWXG8jrz7//HNJ0jPPPGMPTNKl/R86dKisVqs2b97ssE6nTp0c/h5VqlSRv7+/zpw5c8PqBICShpEmAIBTHh4e+uGHH7Rp0yYdPXpU8fHxOnnypAzDkCT7/48ePSpJuvPOO6/YRo0aNa7bzokTJyRJd9xxR57X9/Dw0IwZM/Tcc89p4sSJkqRatWqpZcuW6tq1q+rWrXvd9iUpMDAwV8uVLVv2ituAu7u7q0qVKvr1119ztY38OHHihLy9vXXbbbddMS/nlL5/jo6VL1/+imU9PDxks9luTJEAUAIx0gQAcGrs2LF67LHH9MMPP6hatWqKiIjQu+++qxdeeMFhuaysLElyGNXI4enpmev2MjMzr5iWE8yc6dChg7755htFR0erT58+ysrK0rvvvquePXvq/fffz1Xbl4+mOXP5KM8/68zNNvL7w7jOnoecEPTP599s5q0eAAqKkSYAwDXFxsbq008/1YMPPqi5c+c6hIW9e/c6LJszwnTkyJErbmRw5MiR67Z1++23X3PZY8eOOV03NTVVBw8eVJUqVdSxY0d17NhRkhQXF6eBAwdqwYIFGjhw4HVryK3k5GSlpKTI19fXPi0rK0u///67fT+kSyHsar8Jld9T46pVq6YjR44oPj7+itGmnBGuypUr52vbAIBr4+snAMA1JSUlSbp06tflgencuXNavXq1pL9HTTp27Cg3NzctW7ZMFy9etC+bkJBgv522M/fdd5/8/Pz0wQcfKCUlxT797Nmz2rBhg9N1Dx48qEceeUQLFy50mF6zZk2VKVNGbm5/f0eYMxKUm9Gra7HZbFq+fLnDtPfee08XLlzQgw8+aJ9WoUIFJSUlOZwyl5mZab826XK5qatTp06SpHnz5jksd+HCBS1atEgWi0UdOnTI304BAK6JkSYAwDU1atRIZcuW1aJFi5SRkaFq1arpjz/+0Jo1a3T+/HlJsv//tttu01NPPaXo6Gj169dPYWFh9luG54y2XOu0Nkny8vLSlClTNHbsWPXs2VN9+vSRYRhavny5/aYTzups0aKFVq5cqZSUFDVt2lRWq1WbNm1SfHy8wx31cq5bWrx4se6///58hQwvLy+99dZb+uOPP1SvXj3t2bNH69at0913363Bgwfbl3v44YcVGxurIUOGqH///rLZbFqzZs1Vg1Fu6urevbs+//xzrV+/XidPnlT79u2Vnp6uNWvW6MSJExo3bpyqVq2a5/0BADhHaAIAXFNAQIDeffddzZkzRx999JGysrIUFBSkTp06adCgQercubO+/fZbPfnkk5KkESNGqFy5clq2bJlmz54tf39/hYeHKzMzU0uWLLnq9U6X69Kli/z9/bVgwQItXLhQnp6e6tq1q26//XZNmzbtmuuZTCZFR0fr3Xff1aZNm/TVV19JkoKDgzV79myHW54PGDBAP/74o9asWaOYmJh8hSZfX1+99tprmjFjhjZs2CA/Pz89+uijeuaZZxyu3+rdu7fS0tK0YsUKzZw5U+XKlVP37t3VqlUrPfLIIw7bzE1dFotFCxcu1Hvvvaf169dr9uzZ8vLyUr169TRp0qSr/pAuAKDgTEZBzk8AAOD/S0tLk9VqVZkyZa6YN2nSJH300Uf64osvrvkbRAAAFFdc0wQAKBS//vqrGjdurPnz5ztMP3/+vL788kuVL1/+qrfKBgCguOP0PABAobjnnntUu3Ztvfnmmzp37pzq1KmjpKQkrV27VmfPntV//vMfp9c0AQBQXHF6HgCg0Jw7d06LFy/Wtm3bdOrUKXl5eSkkJESPP/64mjVr5uryAADIF0ITAAAAADjBNU0AAAAA4AShCQAAAACcIDQBAAAAgBO35N3zDMOQzcalXAAAAMCtzGw25erOrrdkaLLZDJ07d8HVZQAAAABwoYCA0rJYrh+aOD0PAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMCJW/LueQAAAEBxZrPZZLVmu7qMm5rF4iazuXDGiAhNAAAAQDFhGIZSUs4pPT3V1aWUCF5ePvL1DcjVbzE5Q2gCAAAAiomcwOTj4y8Pj1IF/rB/qzIMQ1lZmUpNTZQk+fkFFmh7hCYAAOASZrNJZjMfCPPKZjNksxmuLgM3gM1mtQcmHx9fV5dz0/PwKCVJSk1NVJky/gU6VY/QBAAAipzZbJK/v5fMZourS7np2GxWJSamE5xKIKvVKunvD/souJzn0mrNltnske/tEJoAAECRuzTKZNHRTxcp/exJV5dz0/AKrKQ7w56Q2WwiNJVgnJJXeArruSQ0AQAAl0k/e1Lpp0+4ugyg2HPV6aycDnqJy0NTdna2FixYoPXr1yspKUl169bVv//9bzVo0ECSFBcXp+nTp2v//v0KCAjQY489poEDB7q2aAAAAKCImM0mlS3rLYul6H9i1Wq1KSkpLc/B6dSpU9q//yd16NDpBlVWtFwemt544w2tWrVKM2bMUNWqVbVo0SI9/vjj+uyzz+Tu7q5BgwapXbt2mjp1qvbu3aupU6eqdOnSCg8Pd3XpAAAAwA1nNptksZi1YMUOxSckF1m7t1Xw0/B+9+XrdNDp0yerYsVKhKbCsm3bNoWFhally5aSpKioKK1atUp79+7V0aNH5e7urmnTpsnNzU01atTQ8ePH9fbbbxOaAAAAcEuJT0jWsfhEV5eRK4ZRsk7pK/oxvn8IDAzUl19+qT/++ENWq1UffvihPDw8FBwcrNjYWDVt2lRubn9nu9DQUB07dkxnzpxxYdUAAAAArmbEiCe1d++P2rTpU/Xq1VW9enXV/PmvacCA3nroofbas+cHjRjxpKZPn3LFepdPO3bsqMaNe0YPPHC/unfvpKlTJ+rsWddkAJePNE2YMEGjRo1S+/btZbFYZDabFR0drWrVqunUqVOqVauWw/IVKlSQJJ08eVLlypXLd7tubi7PiwAA3LJccW1GScLzVzLZbCXjrnkvvzxL48ePVoUKQRo9eryeeGKg1q79SK++OldlypRR9ep3XXcbZ878peHDH9cDDzyokSPHKD09Xe+++5aGDRus99//UF5eXnmqyWIxFejzv8tD0+HDh1WmTBktWLBAQUFBWrVqlcaNG6dly5YpIyNDHh6O91MvVerSvdYzMzPz3eal34YoXaC6AQAAXMXXN28fGHFzyMiw6MwZ8xUf8F0dkvPafkCAv9zd3eXp6any5QMlSc2b36fmzZvblzGZTDKZHPfz8mkbNqxRhQpBGjduvH3+yy+/qk6d2uvrr79QWFi3XNVis5lkNpvl5+ctT0/PPO3H5Vwamk6ePKmxY8dq6dKlaty4sSSpXr16Onz4sKKjo+Xp6amsrCyHdXLCkre3d77btdkMpaSk5b9wAABQIBaLmQ/+BZCSki6r1ebqMlDIsrIyZbPZZLUays4uPn9fq9WW53oMw5Bh/L0ft91W1WEb/5z/z2m//BKnI0d+U9u29zlsNysrU0eOHMl1PVarIZvNpuTkNKWnW6+Y7+vrlatQ6NLQ9NNPP+nixYuqV6+ew/T69evrm2++UeXKlZWQkOAwL+dxUFBQgdouTgcigFuTq35z42bG74UAl+TnQyyKP6u15L6+5Zwt5ozV+neosdkMNWrUWGPHRl2xnI9PmTy3X9Ag6tLQVLFiRUnSwYMHFRISYp9+6NAh3XHHHapfv75Wrlwpq9Uqi8UiSYqJidGdd96pwMBAl9QMAIXh0mnCXjKbLa4u5aZis1mVmJhOcAKAYs5kcv6loLu7uy5cuGB/bLPZ9Oeff6hKlaqSpOrVa+iLL7aoQoUg++U6KSnJeumlyerbd4AaNWp844q/CpeGppCQEN17772KjIzU5MmTVbFiRa1fv147d+7UihUrVKVKFS1evFgTJkzQ448/rp9//llLly7V1KlTXVk2ABTYpVEmi45+ukjpZ0+6upybgldgJd0Z9kS+fi+kKDBymDeuvkYDwI3l5eWtkyf/VELC6avOv+eeEK1c+YFiYv5PVapU1YcfLtf586n2+T169NKGDWs1bdpEPfro45KkBQte02+/Hdadd9Yokn24nEtDk9ls1htvvKHXXntNzz33nJKTk1WrVi0tXbpU9evXlyQtXrxY06dPV48ePVS+fHmNHz9ePXr0cGXZAFBo0s+eVPrpE64uAwVkNptUtqw3QQDADXVbBb+bpr2HHw7X9OmT9eij/a56p7u+ffsrPv4PTZoUJQ8Pdz30UHd16NDR/vtOlSvfpvnz39Kbb87X008PkcViUb169TVv3pvy9/fPd135ZTJK2i9P5YLVatO5cxeuvyAA3CBubmb5+5fWgfemEZpyySuomuo++oISEy8Uu2s5cv6eC1bsUHxCsqvLuSnUr11ZfTo3oA/kUXHuByi4ixezdPbsSQUGVpK7+993kHblFzNWq01JSWnFcoQ/N671nOYICChd/G8EAQBASRKfkKxj8YmuLuOmULm8r6tLAG4aNpuhpKQ0l5wCzA14LiE0AQAAAMUc4cW1OPkaAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAn+J0mAAAAoJgzm038uK0LEZoAAACAYsxsNsnf30tms6XI27bZrEpMTC+2walXr6568MEwDRky9Ia2Q2gCAAAAirFLo0wWHf10kdLPniyydr0CK+nOsCdkNpuKbWgqKoQmAAAA4CaQfvak0k+fcHUZtyRCEwAAAIBC07JlY40ePV6bN3+mw4cPqUqVqnryyafVsmVrSdI777ylPXt+UGBgoHbu/D89+OBDGj16vPbt+0lvvjlfcXEHVLZsWd13XysNGzZcpUv7SJJSU1P12muz9N13X8vNzU0DBjxWZPvE3fMAAAAAFKo335yvTp26aOnS5WrevKWef/7f2rfvJ/v8vXt/VEBAOS1Z8oF69eqrw4d/1bPPPq1mzZrrvfdWaPLk6Tp4ME6jR4+QYVw6NfCFF6IUF/c/vfrqXM2du0A7d+7QqVNFc7oioQkAAABAoerSJUzh4f9StWp36KmnRio4uK5Wr/7QYZkhQ4bqttuqqGrValqx4n01bRqqgQMHq2rVaqpfv4GmTJmuAwf2a8+eH3TixDHt3h2j0aPHq379hqpZs7YmT35JHh4eRbI/nJ4HoMBcdRvUm5nFwndWAICSq1Gjxg6P69UL0e7dMfbH/v4B8vHxsT8+ePCg/vjjhB544P4rtnX8+DElJydJkurUqWufHhAQqMqVbyvkyq+O0ASgQMxmk8qW9SYEAAAAO4vFMWZYrTaHW6aXKlXKYb5h2NSx44MaOHDwFdsqW9ZfsbG7JOmKu/j9s50bhdAEoEDMZpMsFrMWrNih+IRkV5dz06hfu7L6dG7g6jIAALghfvnlgFq2bGV/vH//z6pdO/iay995Zw0dPXpEVapUtU87fvyYFix4XcOGDVfNmrUlSfv2/aQWLVpKks6fP6/4+N9v0B44IjQBKBTxCck6Fp/o6jJuGpXL+7q6BAAAbpiPPlqhatXuUHBwHX388TodPnxIUVGTrrl8374DNHz44/rPf15VePi/lJp6Xv/5zwxlZmaqatXb5e7urrZtO2ju3Jlyd3dXYGCg3nxzgS5evFgk+0NoAgAAAG4CXoGVbpr2Hn64pz76aLmOHDmsGjVqas6c+brrrprXXP6ee+ppzpz5Wrz4DQ0ePEDe3l66994mGj78Wbm7u0uSJk6covnzX9fkyc/LZrOpe/eeSkoqmi9sCU0AAABAMWazGbLZrLoz7AkXtG294jqi3Ljjjup6+ulRV503ZMhQDRky9Irp997bRPfe2+Sa2yxVylNjx0Zq7NjIPNdTUIQmAAAAoBiz2QwlJqa75E61lwJb3kNTSUNoAgAAAIo5wotrEZoAAAAAFJrvvot1dQmFjh9WAQAAAAAnCE0AAAAA4AShCQAAAChGDINrlwpLYT2XhCYAAACgGLBYLJKkrKxMF1dScuQ8lxZLwW7lwI0gAAAAgGLAbLbIy8tHqamXfrDVw6OUTKaiv814SWAYhrKyMpWamigvLx+ZzQUbKyI0AQAAAMWEr2+AJNmDEwrGy8vH/pwWBKEJAAAAKCZMJpP8/AJVpoy/rNZsV5dzU7NY3Ao8wpSD0AQAAAAUM2azWWazh6vLwP/HjSAAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThSL0LR+/Xp16dJF9erV00MPPaRNmzbZ5/3xxx8aOnSoGjVqpJYtW+q1116T1Wp1YbUAAAAAbiUuD00bNmzQhAkT1L9/f23cuFFhYWEaM2aM9uzZo4sXL2rIkCGSpJUrV2rKlClasWKFFixY4OKqAQAAANwq3FzZuGEYev311zVw4ED1799fkvTUU08pNjZWu3fvVnx8vP7880999NFH8vPzU61atXT27FnNnDlTw4YNk4eHhyvLBwAAAHALcOlI09GjRxUfH6+uXbs6TH/nnXc0dOhQxcbG6u6775afn599XmhoqFJTUxUXF1fU5QIAAAC4Bbl0pOno0aOSpLS0NA0ZMkQHDhxQlSpV9NRTT6ldu3Y6deqUKlas6LBOhQoVJEknT55U/fr18922m5vLz0wESgSLhb6EolUcj7niWBNKNo45oGi5NDSlpqZKkiIjIzVixAiNGzdOmzdv1tNPP60lS5YoIyNDvr6+DuuUKlVKkpSZmZnvds1mk/z9S+e/cACAy/j6erm6BMDl6AdA0XJpaHJ3d5ckDRkyRD169JAk1alTRwcOHNCSJUvk6emprKwsh3VywpK3t3e+27XZDKWkpOV7fQB/s1jMvHmjSKWkpMtqtbm6DAf0AxS14tgPgJuRr69XrkZuXRqagoKCJEm1atVymH7XXXfpq6++UtOmTXXo0CGHeQkJCQ7r5ld2Ni80AHAzslptvIbjlkc/AIqWS0+Ivfvuu1W6dGn99NNPDtMPHTqkatWqqUmTJjpw4ID9ND5JiomJUenSpRUcHFzU5QIAAAC4Bbk0NHl6eurxxx/XggUL9Omnn+rEiRN64403tGPHDg0aNEgdOnRQ+fLl9eyzz+qXX37Rtm3bNGfOHA0ePJjbjQMAAAAoEi49PU+Snn76aXl5eWnu3Lk6ffq0atSooejoaDVr1kyStHjxYk2dOlX/+te/5Ofnp0ceeURPP/20i6sGAAAAcKtweWiSpEGDBmnQoEFXnXf77bfr3XffLeKKAAAAAOASbvIPAAAAAE4QmgAAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtAEAAAAAE645XWF5557LtfLmkwmvfzyy3ltAgAAAACKjTyHplOnTunAgQNKTk7WbbfdpqCgICUlJen48eMyDEMVK1a0L2symQq1WAAAAAAoankOTV26dNGvv/6q5cuXq1GjRvbpR44c0VNPPaVHHnlEjz76aKEWCQAAAACukudrmt58802NGzfOITBJUvXq1fXss8/qnXfeKbTiAAAAAMDV8hyazp07Jz8/v6tvzGzW+fPnC1wUAAAAABQXeQ5N9evX1/z585WYmOgwPSEhQdHR0WrZsmWhFQcAAAAArpbna5qioqI0YMAAtWvXTg0bNpS/v7/Onj2rPXv2KDAwUM8///yNqBMAAAAAXCLPI03BwcHauHGj+vbtq9TUVO3fv18ZGRkaPHiw1q5dq0qVKt2IOgEAAADAJfI80iRJQUFBioyMLOxaAAAAAKDYyVdoysrK0urVq/V///d/+uuvv/Tyyy9r9+7duvvuuxUSElLYNQIAAACAy+Tr7nnh4eGaPn26jh8/rp9//lkZGRn66quvFBERoT179tyIOgEAAADAJfIcmmbOnKkLFy7os88+07p162QYhiRp3rx5qlevnubNm1foRQIAAACAq+Q5NH355ZcaNWqUbr/9dplMJvv0UqVKafDgwfrf//5XqAUCAAAAgCvlOTRlZmaqbNmyV51nsVh08eLFgtYEAAAAAMVGnkNTvXr1tHz58qvO++STT3TPPfcUuCgAAAAAKC7yfPe8UaNG6bHHHlP37t3VunVrmUwmffrpp4qOjtZ3332nxYsX34g6AQAAAMAl8jzS1LhxYy1ZskReXl5avHixDMPQ0qVL9ddff+mtt95SaGhovos5evSoGjZsqLVr19qnxcXFacCAAWrQoIHatWun999/P9/bBwAAAIC8yvNI086dO9WwYUOtXLlSGRkZSk5Olo+Pj0qXLl2gQi5evKhx48YpLS3NPi0xMVGDBg1Su3btNHXqVO3du1dTp05V6dKlFR4eXqD2AAAAACA38jzSNHLkSG3ZskWS5OnpqaCgoAIHJkmKjo6Wj4+Pw7SPPvpI7u7umjZtmmrUqKHw8HA99thjevvttwvcHgAAAADkRp5Dk6+vrzw9PQu1iO+//14ffvihZsyY4TA9NjZWTZs2lZvb3wNioaGhOnbsmM6cOVOoNQAAAADA1eT59LyhQ4fqpZde0tGjRxUcHCxvb+8rlmnSpEmut5eSkqLx48dr4sSJqlSpksO8U6dOqVatWg7TKlSoIEk6efKkypUrl9fy7dzc8pwXAVyFxUJfQtEqjsdccawJJRvHHFC0chWaMjMzVapUKUnS5MmTJUlz586VJIcfuDUMQyaTSXFxcbkuYMqUKWrYsKG6du16xbyMjAx5eHg4TMupIzMzM9dt/JPZbJK/f8FPKQQAFD1fXy9XlwC4HP0AKFq5Ck3t2rXT/Pnz1bBhQzVp0kS9e/dWxYoVC9z4+vXrFRsbq08++eSq8z09PZWVleUwLScsXW2EK7dsNkMpKWnXXxDAdVksZt68UaRSUtJltdpcXYYD+gGKWnHsB8DNyNfXK1cjt7kKTefPn1dCQoKkS9cZ/fvf/1ZISEjBKpS0Zs0anT17Vm3atHGYPnnyZH322WeqWLGivd0cOY+DgoIK1HZ2Ni80AHAzslptvIbjlkc/AIpWrkJTvXr1NHbsWL366qsyDEPDhw+/4rS5HCaTSdu2bctV47Nnz1ZGRobDtI4dO+qZZ55Rt27dtGHDBq1cuVJWq1UWi0WSFBMTozvvvFOBgYG5agMAAAAACiJXoWnOnDlaunSpkpKStG7dOtWtW1cBAQEFbvxao0WBgYEKCgpSeHi4Fi9erAkTJujxxx/Xzz//rKVLl2rq1KkFbhsAAAAAciNXoSkoKEiRkZGSpF27dmn06NEKDg6+oYVJl8LT4sWLNX36dPXo0UPly5fX+PHj1aNHjxveNgAAAABI+bjl+Pbt229EHXYHDx50eBwSEqIPP/zwhrYJAAAAANfCTf4BAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtAEAAAAAE4QmgAAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJxwc3UBNzuz2SSz2eTqMm46Npshm81wdRkAAADAdRGaCsBsNqlsWW9ZLAzY5ZXValNSUhrBCQAAAMUeoakAzGaTLBazFqzYofiEZFeXc9O4rYKfhve7T2azidAEAACAYo/QVAjiE5J1LD7R1WUAAAAAuAE4rwwAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADghMtDU1JSkl544QW1atVKjRo1Ur9+/RQbG2ufv3PnTvXs2VP169dX586dtXHjRhdWCwAAAOBW4/LQNGbMGO3Zs0dz5szRmjVrVKdOHQ0ZMkRHjhzRb7/9pqFDh+r+++/X2rVr1bt3b40fP147d+50ddkAAAAAbhFurmz8+PHj2rFjh5YvX657771XkjRp0iR9++23+uSTT3T27FnVrl1bo0ePliTVqFFDBw4c0OLFi9W8eXNXlg4AAADgFuHSkSZ/f3+9/fbbqlevnn2ayWSSyWRSSkqKYmNjrwhHoaGh+uGHH2QYRlGXCwAAAOAW5NKRJl9fX7Vu3dph2ubNm3X8+HE9//zzWrdunSpWrOgwv0KFCkpPT1diYqICAgLy3babW8HzosXi8rMbb2o8fyUDf0cUteJ4zBXHmlCyccwBRculoemffvzxRz333HPq2LGj2rRpo4yMDHl4eDgsk/M4Kysr3+2YzSb5+5cuUK0oOF9fL1eXAOAmxGsHQD8AilqxCU3btm3TuHHj1KhRI82ePVuSVKpUqSvCUc5jL6/8v1jYbIZSUtLyX+z/Z7GYedEqgJSUdFmtNleXgQKiH6CoFcfXDvoBilpx7AfAzcjX1ytXI7fFIjQtW7ZM06dPV+fOnfXqq6/aR5MqVaqkhIQEh2UTEhLk7e2tMmXKFKjN7GxeaFzNarXxdwCQZ7x2APQDoKi5/ITY5cuX68UXX1T//v01Z84ch9PxGjdurN27dzssHxMTo0aNGslsdnnpAAAAAG4BLh1pOnr0qF5++WU98MADGjp0qM6cOWOf5+npqYiICPXo0UOzZ89Wjx499PXXX+vzzz/X4sWLXVg1AAAAgFuJS0PT5s2bdfHiRW3dulVbt251mNejRw/NmDFDCxcu1KxZs/Tee++pSpUqmjVrFr/RBAAAAKDIuDQ0DRs2TMOGDXO6TKtWrdSqVasiqggAAAAAHHFhEAAAAAA4QWgCAAAAACcITQAAAADgRLH4nSYAAADc/Mxmk8xmk6vLuKnYbIZsNsPVZeA6CE0AAAAoMLPZpLJlvWWxcCJTXlitNiUlpRGcijlCE1yGF9W84ZsoAEBxZjabZLGYtWDFDsUnJLu6nJvCbRX8NLzffTKbTbzHF3OEJhQ5vzKeMmw2+fp6ubqUm4rNZlViYjovqgCAYi0+IVnH4hNdXQZQqAhNKHKlPT1kMpt19NNFSj970tXl3BS8AivpzrAn+CYKAADABQhNcJn0syeVfvqEq8sAAABwKS5ZyLuivmyB0AQAAAC4AJcs5F9RX7ZAaAIAAABcgEsW8scVly0QmgAAAAAX4pKF4o8TKAEAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnCA0AQAAAIAThCYAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHCC0AQAAAAAThCaAAAAAMAJQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtAEAAAAAE4QmgAAAADACUITAAAAADhBaAIAAAAAJwhNAAAAAOAEoQkAAAAAnLgpQpPNZtO8efN0//33q0GDBnriiSf0+++/u7osAAAAALeAmyI0LVy4UMuXL9eLL76olStXymaz6fHHH1dWVparSwMAAABQwhX70JSVlaV3331XzzzzjNq0aaPg4GDNnTtXp06d0pYtW1xdHgAAAIASrtiHpl9++UUXLlxQ8+bN7dN8fX1Vt25dff/99y6sDAAAAMCtwGQYhuHqIpzZsmWLRo4cqZ9++kmenp726aNGjVJGRobeeuutPG/TMAzZbAXfbZNJMpvNSk7NkNVqK/D2bhUe7hb5eJfSxQspMmxWV5dzUzCZLXIv7Subzabi1mPpB/lDP8g7+kHJQh/IH/pByUI/yJ/C7Adms0kmk+m6y7kVrJkbLz09XZLk4eHhML1UqVJKTk7O1zZNJpMslus/Obnl5+N5/YVwBffSvq4u4aZjNhffwWH6Qf7QD/KOflCy0Afyh35QstAP8qco+0Hx7XH/X87o0j9v+pCZmSkvLy9XlAQAAADgFlLsQ1OlSpUkSQkJCQ7TExISFBQU5IqSAAAAANxCin1oCg4Olo+Pj3bt2mWflpKSogMHDqhJkyYurAwAAADAraDYX9Pk4eGhAQMGaPbs2QoICNBtt92mWbNmqWLFiurYsaOrywMAAABQwhX70CRJzzzzjLKzszVx4kRlZGSoSZMmeuedd+Tu7u7q0gAAAACUcMX+luMAAAAA4ErF/pomAAAAAHAlQhMAAAAAOEFoAgAAAAAnCE0AAAAA4AShCQAAAACcIDQBAAAAgBOEJgAAAABwgtCEGyYiIsL+79q1a7uwEsC15s2bp9jY2ELf7ooVK7RixYp8z0fxVZjHTFRUlNauXXvF9LVr1yoqKqpQ2sDVde/e3dUloJjh89DNy83VBaDk2r17t6tLAIqF77//Xs2aNSv07fbr169A81F83ahjBkVrw4YNri4BQCEhNKFQvPnmm/r4449lsVh03333KT09XZLUs2dP+zecU6dO1Z49e5SZmalXX31VISEhOnHihKZMmaLExER5eHgoMjJSjRo1UlRUlBITE3XixAmNGjVKnTt3duXu4SY3Z84cbd68WRaLRd27d1fHjh31wgsvKCkpSd7e3powYYJCQkIUFRUlT09P7d27V0lJSRo9erS2bdumuLg4tW3bVhMmTNDatWu1efNmpaamKiEhQa1bt9aECRMUHx+vgQMHavv27ZIufYu/e/duNWnSRPv379fEiRM1b948lS5dOs/H/IwZMxQQEKAnn3xSkhQZGammTZvqzz//lCQNHz7c3r8sFovatWunkSNHKjo6WpI0cuRIffnll3rttddks9lUtWpVTZs2TeXKlVO7du3UvXt37dixQ0lJSZo0aZLuv//+Iv4LlXy7du3SG2+8oTJlyui3335TxYoVNWfOHH366adav369MjIyJF06Vvft2+dwzLz88ssaMWKEPUTVrl1bBw8eVHR0tPbu3atTp06pd+/eqlu3rubMmaPMzEwlJydr7Nixeuihh3JV36ZNm7RkyRJlZGQoIyND06ZNU2hoqCIiIlS/fn3FxsYqISFBI0eOVI8ePZSamqrnn39ev/76q8qXLy+TyaSnn35akjR//nz997//lSSHY3DZsmVX7GutWrUUGxuradOmyWw2q3Hjxvr666+1detWnTt3Ti+88IL9OB8xYoTatWt3Re0RERG68847tX//fqWnpysqKkqtW7e+4vlp3br1Vfv9yZMn9dxzz+nMmTPy8PDQlClTFBISog0bNui9996T1WrVXXfdpalTp8rb2/uqfW3v3r166aWXZBiGSpUqpZdeeknVq1d3+FudPn1aJ06cUHx8vL0WSXr99de1ceNGlSlTRjVq1FDVqlU1cuTI/B5qcKFdu3ZpwYIFcnNz07Fjx3T//fcrKChI27Ztk81m09tvv21fNj09XRMnTtTBgwdlMpk0ZMgQPfzww9d8j8nMzNT48eN14sQJmUwm9enTR3379nXh3t6CDKCAvvrqKyM8PNxIS0szLl68aAwbNsxYtmyZUatWLfsytWrVMjZu3GgYhmG89957xsiRIw3DMIy+ffsaP//8s2EYhnH8+HGjbdu2xsWLF43IyEhj7NixRb8zKHE2b95s9OnTx8jIyDAyMjKMXr16GU2aNDE+++wzwzAMY8+ePUabNm2MzMxMIzIy0hg2bJhhGIaxdu1a49577zXOnDljnD9/3mjYsKGRnJxsrFmzxggNDTUSEhKMzMxMo0+fPsZnn31m/P7770bbtm3t7a5Zs8aIjIw0DMMwBgwYYMTExBiGkb9jPi4uzujWrZthGIaRkZFhtGjRwjh//rwxb948Y968eUZcXJzRo0cP+/wxY8YYaWlp9vlnzpwx7rvvPuPEiROGYRjGokWL7H2wbdu2xjvvvGMYhmFs2bLFvh0UrpiYGKNBgwZGfHy8YRiGMWzYMGPp0qVGRESEkZaWZhiGYbz++uvGtGnTDMNwPGYu/7dhGPbX1nnz5hn9+vWzTx85cqRx6NAhwzAMY+fOnUZYWJhhGIYRGRlprFmz5oqaco5Rq9VqREREGGfOnDEMwzBWr15tDB061N52Tk3/+9//jKZNmxqGYRgzZswwXnrpJcMwDOPEiRNGgwYNjJiYGCMmJsYYMGCAvY2cY/D8+fNX3desrCyjVatW9j6xaNEiez8aM2aMsXnzZsMwDOPs2bNGhw4d7DVebsCAAcb48eMNm81mHDhwwAgNDTUyMzOveH7Cw8Ov2u+HDh1qLF261DAMw9i1a5cxZMgQ4/Dhw0bfvn2N9PR0wzAMY+HChcaMGTOu2deefvpp44svvjAMwzA2btxorF279oq/Vc+ePY3MzEwjNTXVaNmypfHLL78Y27dvN3r37m2kp6cbaWlpRs+ePY158+ZdsY+4OVzez9PS0owGDRoYK1asMAzDMKKiooylS5faj4lXX33VmDp1qmEYl47vdu3aGXFxcdd8j9m6dasxYsQIwzAM49y5c8a4ceNcs5O3MEaaUGAxMTEKCwuTl5eXJCk8PFzr16+/YrmOHTtKkmrVqqWtW7fqwoUL2rdvnyZOnGhfJjs7WydPnpQkNWzY8MYXjxJv165devDBB1WqVClJ0tKlS9WmTRs9+OCDkqQGDRrIz89PR44ckSS1adNGklS5cmXVrFlTgYGBkqSyZcsqJSVFktS2bVuVL19ektSlSxd9//33qlev3nVrye8xHxwcLEn67bffdOjQIYWGhsrHx8c+v1q1asrKylL//v3VunVrjR492t4fJennn39WSEiIqlatKknq06ePwzeerVu3treTlJR03f1A/tSsWVOVK1eWJNWpU0fnz5/X3Llz9dlnn+nYsWP69ttvVadOnTxts0GDBvZ/z5o1S19++aW2bNmin376SRcuXMjVNsxmsxYuXKjt27fr6NGj2r17t8zmvy95zjk+6tSpYz8+vvvuO82aNUuSVLVqVbVo0cJpGz4+Plfd10OHDikgIMDef/r06aPly5fb2/j111+1YMECSZf6ym+//Wbvk5fr3bu3TCaT6tSpo4oVK+rgwYMOz8+FCxd0/Pjxq/b7Xbt22feladOmatq0qZYtW6bjx4+rT58+9rarVq16zb7Wrl07TZw4UW3btlXbtm3VqVOnK2ps3ry5PDw85OHhodtvv13JycnasWOHwsLC5OnpKUnq1q2b/XUGN6fatWvb+7m/v7+aN28u6dJ7yuV/25iYGE2fPl2SFBAQoPbt22v37t3y8fG56nvM0KFDNX36dA0ZMkStW7dWZGRkEe8ZCE0oMJvNdsW07OzsK6a5uV063Ewmk309Dw8Ph3O+T58+bX+huPxDH5BfFovFfsxJUnJy8hXLGIZhP2bd3d3t03OO2X+6fLrNZpPZbJbJZJJhGPbpFy9evGK93B7zlwere+65R9OnT1f37t21adMmHTx4UL169XLYrre3t9avX69du3bpu+++U9++fe2nR+W0+8/9vby+nEB5+fOEwpfzPEuXnus///xTvXv3VkREhFq1aqVy5copLi7uquvmHFtZWVkO0y9/nXzkkUfUtGlThYaGqnnz5ho3bpzDsitWrNDKlSslSX379rXXc+HCBYWHh6tbt25q0qSJateurQ8++OCKui8/PiwWi8Pxfvl+/bMfuLu76+TJk+rfv/8V+2qxWK76HiJdOm7ff/99lS1bVpKUkJCggIAAPfHEE0pISJAke/i3WCwO6+U8znl+rlZrTr93c3Nz2Ldff/1VVqtVXbp0sffDtLQ0ZWVlXbOvhYeHq3nz5vrqq6+0dOlSffXVV3rppZcc2vvn398wDJnN5mvuP25Ol7+HSI7H5uX+eUxe/j50tfeYoKAgbdq0STt27NC3336rHj16aOPGjfL19S3kPcC1cPc8FFhoaKg+/fRTpaenKzs7W2vWrFGTJk1ksViuGp5ylClTRnfccYf9A2RsbKx69uzpdB0gr5o2baqtW7cqKytLWVlZGjZsmC5cuKBNmzZJkvbu3auEhATVqlUr19v89ttvlZKSoszMTG3cuFEtW7aUn5+fkpKSlJCQIKvVqi1bttiXt1gsslqtuT7m69Wrpw0bNmjDhg32byK7du2qzZs3Ky4uTi1btnRYPjY2Vk888YRCQ0MVGRmpGjVq6OjRo/b59evX188//6zff/9dkvThhx+qadOmeXgWcSPs27dPd9xxhwYNGqT69evrm2++kdVqlfT3MSNd+rb6l19+kSR9/vnnV91WUlKSjh07pmeffVatW7fWjh077Ovn6Nevn/24uvwmIceOHZPJZNJTTz2l0NBQhzqu5b777tO6deskSadOndKuXbtkMpnk7++vY8eOKT09Xenp6frqq6+c7mv16tV1/vx5/e9//5Mkh7MUQkND7aNOx44dU1hYmJKTk7Vo0SL7fgQFBUmSNm7caG8nKSnpiv7s4+OjqlWrXrXfN23a1L7+nj17NGbMGDVr1kxbt27VmTNnJEmvvPKKFi5ceM2+9sQTT+jo0aN65JFHNGrUKB04cMDp83f587hp0yZlZmYqKytLmzZt4suLW0RoaKhWrVolSTp37py2bdumxo0bS7r6e8wnn3yiKVOmqH379po4caK8vb3tZymgaDDShAJr27at4uLi1KtXL2VnZ6tFixYaOHCgfv75Z3Xr1k2rV6++5rqzZs3SlClTtHjxYlksFr3++uvy8PAowupR0nXo0EEHDhxQeHi4bDabwsPD1bp1a02ZMkULFy6Uu7u7oqOj83TclS9fXkOHDtW5c+cUFhZmP6Vv2LBh6tu3r8qVK6fGjRvr3Llzki6d8jd58mS98sor+T7my5cvr6CgINWoUeOKby7vvfdeVa9e3X6aT926ddWqVSv7B9Fy5cpp2rRpGjFihLKzs1WxYkW9/PLLud5f3BgtW7bUL7/8oi5dusjDw0MhISE6dOiQJMdjZujQoYqMjNS6devUvHlz+8jk5cqWLavevXvroYceko+Pj+rXr6+MjIxcnaIXHBysunXr6sEHH5Snp6eaNGmi+Pj4q47O5Hjqqac0adIkde3aVeXLl1flypXl6empmjVrqmPHjgoLC1NQUJDuvfdeSZfCwYoVK67YVw8PD82ZM0eTJk2SYRgKDg62n6o2ceJETZ48WV27dpVhGJo+ffpVT82TLo3Y9ujRQzabTXPmzLnqKHFO3/tnv580aZImTpyo5cuXy8PDQ6+++qqCg4M1YsQIDRo0SDabTTVq1FBUVJS8vb2v2tcCAwM1depUzZ49W25ubrm+lXvr1q21b98+9ejRQ6VLl5a/v7/DiBRKruHDh2vKlCkKCwuT1WrVk08+qZCQEB0+fPiq7zGZmZn64osv9NBDD8nd3V2dOnXi9uVFzGQ4e1UEADjIuSvejBkzXF0K4DIff/yxKlasqKZNmyo1NVU9e/bUqlWr5Ofnl6ftGIahmTNnavjw4fLx8dG2bdv08ccfa968ebneRkREhMPdBW8mP/30kw4dOqTevXvLMAw988wzCg8Pt38Rg1sP7zHFFyNNAAAgT6pXr67JkyfbT+MbNWpUngOTdOnansDAQP3rX/+Su7u7AgMD9eKLLxZ2ucXWHXfcoQULFuj999+XdGmEkcAEFE+MNAEAAACAE9wIAgAAAACcIDQBAAAAgBOEJgAAAABwgtAEAChxuFwXAFCYCE0AgBLliy++UGRkpKvLAACUINxyHABQoixdutTVJQAAShhGmgAAAADACX6nCQBQYkRERGj37t32x++//778/Pw0f/58xcbG6vz58woICFCnTp00btw4eXp6SpJSU1M1c+ZMbd26VRkZGWrTpo3q16+vV155RQcPHnTV7gAAiglCEwCgxDh8+LD+/e9/S5ImT56s8uXLq1u3bmrQoIEiIiLk4eGhb775RkuWLNHYsWP15JNPSpIGDhyouLg4jR49WpUrV9by5cu1c+dOZWVlEZoAAFzTBAAoOe666y75+PhIkho0aKDvvvtOderU0euvv26f3qJFC+3YsUO7du3Sk08+qZ07d2rXrl2Kjo5Wx44dJUmtWrVSWFiYfvvtN5ftCwCg+CA0AQBKrJYtW6ply5a6ePGiDh8+rOPHj+vQoUM6d+6cypYtK0mKiYmRu7u7OnToYF/PbDarS5cuio6OdlHlAIDihNAEACixbDab5syZow8++EBpaWmqVKmSQkJCVKpUKfsyiYmJKlu2rMxmx3sjBQYGFnW5AIBiitAEACix3n77bS1dulRTp05Vx44dVaZMGUlSr1697MsEBQUpMTFRNpvNITidPXu2yOsFABRP3HIcAFCiXB58fvjhB911110KDw+3B6bTp0/r0KFDstlskqSmTZsqOztb27dvt69nGIa2bdtWtIUDAIotRpoAACWKr6+v9uzZo507d+r222/Xd999p7ffflsNGjTQ8ePH9dZbbykrK0vp6emSpCZNmui+++7ThAkTdObMGVWuXFmrV6/WwYMHZTKZXLw3AIDigFuOAwBKlJiYGD333HP666+/9OKLL2rfvn3asmWLzp8/r0qVKumhhx6SyWTSW2+9pR07dsjX11fJycmaMWOGtm3bpuzsbLVv316+vr5av369fvzxR1fvEgDAxQhNAIBbWnx8vPbu3av27dvbf+xWkp555hn9/vvvWrdunQurAwAUB5yeBwC4pZnNZkVFRal9+/bq1auXLBaLvv32W23ZskWvvPKKq8sDABQDjDQBAG55MTExWrBggeLi4pSdna0aNWpo0KBBCgsLc3VpAIBigNAEAAAAAE5wy3EAAAAAcILQBAAAAABOEJoAAAAAwAlCEwAAAAA4QWgCAAAAACcITQAAAADgBKEJAAAAAJwgNAEAAACAE4QmAAAAAHDi/wHxKTsvTvZRlgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Few-shot with GPT 3.5\n",
    "method = \"few_shot\"\n",
    "model = \"gpt-3.5-turbo-0613\"\n",
    "y_pred[method][model], performance[method][model] = evaluate(\n",
    "    test_df=test_df, model=model, system_content=system_content,\n",
    "    assistant_content=assistant_content, tags=tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e59a3b9-69d9-4bb5-8b88-0569fcc72f0c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 191/191 [02:13<00:00,  1.43it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"precision\": 0.9407759040163695,\n",
      "  \"recall\": 0.9267015706806283,\n",
      "  \"f1\": 0.9302632275594479\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA00AAAE9CAYAAADXvonEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8h0lEQVR4nO3de3zP9f//8fv7/d5mm9lsw0ZOJYwyh5iRMwmRGB8qU6gop8KnrZBDKSHK8CmHqI9Qcqokh6SDn9GK8LEcMmRhH2xjdrK9X78/fL0/3o23bWbvbW7Xy6VL9jo9H6/3Xs+9d9/z9Xq+TYZhGAIAAAAAXJfZ2QUAAAAAQFFGaAIAAAAABwhNAAAAAOAAoQkAAAAAHCA0AQAAAIADhCYAAAAAcIDQBAAAAAAOEJoAAAAAwAFCEwAAAAA4QGgCgDtQVFSUateunav/IiMjnV2uZs2apdq1a2vnzp2SpJMnT6p27doaM2ZMvo535swZpaam3nS767UTHh6u2rVrKysrK19t56WuyMhI1a5dW8ePHy/wtgAAuefi7AIAAIXvoYceUtWqVe2WvfXWW0pMTNS0adPslv99u6LAz89P06ZNU5UqVfK87+eff64pU6boyy+/lKen521rpyDq6tOnj5o1a6Zy5crd9vYBADdGaAKAO1BQUJCCgoLslr333ntKTExU9+7dnVRV7nl6eua7zujo6FyNMt1qO3l1vboaNmyohg0bFkr7AIAb4/Y8AAAAAHCA0AQAuKmNGzdq4MCBatq0qe677z41bdpUQ4YM0f79+3Ns+8UXX6hnz55q0KCBWrZsqRkzZmjlypV2zyTdrK3evXurQYMGatWqlWbPnq3s7Gy7ba73rFFaWpreeustderUScHBwWratKkGDx6sX375xbZNu3bt9OWXX0qS2rdvr/DwcElXnh2qV6+evv/+e7Vt21b16tXTqFGjHD47deDAAYWHhys4OFjNmzfXq6++qoSEBLtt2rVrp1atWuXY9+rrsXr16pvW9fdnmrKzs/Xvf/9b3bt3V3BwsBo1aqT+/fvr+++/t2tj9erVql27tnbs2KFp06apTZs2uv/++9WpUyctXrz4Jt8FAMC1uD0PAODQkiVL9NZbb6lp06YaNmyYXF1dtX//fq1du1a7du3Sli1b5OfnJ0maP3++3nnnHd1333168cUXdfHiRS1dujTXbS1btkyTJk1SrVq1NHLkSKWmpmrZsmVKS0u76b6jRo3S9u3b9eSTT+qee+7R2bNn9cknn+ipp57S559/rqCgIL366qtauHChdu/erVdeeUU1a9a07Z+VlaUxY8aoX79+Klu2rAIDAx2299RTT6lZs2aKiIjQwYMHtXLlSkVHR2vNmjXy8fHJ9TlLcljXtaxWq4YNG6atW7eqadOmGj16tC5duqTVq1frueeeU2RkpAYMGGC3z9ixY+Xp6an+/fvLxcVFy5Yt09SpU+Xl5aXevXvnqU4AuFMRmgAAN5Sdna33339fderU0eLFi2WxWGzrvL29tWjRIu3atUudOnXSmTNnFBUVpfvuu08rVqyQm5ubJKl79+7q1q3bTdtKSUnR9OnTVbNmTX322Wfy8PCQJPXs2fOmzxWdP39eW7du1eOPP66IiAjb8tDQUEVGRmrfvn0KCgpShw4d9M0332j37t3q0KGDKleubNvWarWqX79+GjlypG3ZyZMnb9jmY489pgkTJti+rlmzpt544w0tWrRIo0aNuun5XstRXdf64osvtHXrVj322GOaOnWqTCaTJKl///4KCwvT9OnT1b59e7vJO0qXLq1Vq1bZvh/t2rVT+/bttWrVKkITAOQSt+cBAG7IYrHohx9+0EcffWQXmFJTU+Xq6irpStiRpC1btigzM1MDBw60/YIuXZl979FHH71pWzt27FBqaqp69eplC0ySVLFiRXXt2tXhvl5eXipTpow2btyolStX6r///a+kKxMpXL3dLzeaN2+eq+0kaejQoXZf9+3bV2XKlNGmTZtyfYy8+uabbyRJI0aMsAUm6cr5Dx48WNnZ2dq4caPdPg8//LDd96Ny5cry9fXV2bNnb1udAFDSMNIEAHDIzc1Nv/zyizZs2KC4uDjFx8fr1KlTMgxDkmz/j4uLkyTdfffdOY5Ro0aNm7Zz4sQJSVL16tXzvL+bm5umTp2qV155RePGjZMk1apVSy1atFC3bt1Ut27dm7YvSf7+/rnarmzZsjmmAXd1dVXlypV1+PDhXB0jP06cOCFPT0/dddddOdZdvaXv76Nj5cuXz7Gtm5ubrFbr7SkSAEogRpoAAA6NHj1aTz/9tH755RdVrVpV4eHh+vDDD/Xaa6/ZbZeZmSlJdqMaV7m7u+e6vYyMjBzLrgYzRzp06KAffvhBUVFR6tOnjzIzM/Xhhx+qZ8+e+vjjj3PV9rWjaY5cO8rz9zpzc4z8fjCuo9fhagj6++tvNvNWDwC3ipEmAMANxcTE6KuvvlLnzp01a9Ysu7CwZ88eu22vjjAdPXo0x0QGR48evWlb1apVu+G2x44dc7hvSkqKDh48qMqVK6tjx47q2LGjJCk2Nlb9+/fX3Llz1b9//5vWkFvJycm6cOGCvL29bcsyMzP1559/2s5DuhLCrveZUPm9Na5q1ao6evSo4uPjc4w2XR3hqlSpUr6ODQC4Mf78BAC4oaSkJElXbv26NjCdP39en3/+uaT/jZp07NhRLi4uWrp0qS5fvmzbNiEhwTadtiMPPvigfHx89Mknn+jChQu25efOndO6desc7nvw4EE98cQTmjdvnt3ymjVrqkyZMnJx+d/fCK+OBOVm9OpGrFarli1bZrfso48+0qVLl9S5c2fbsgoVKigpKcnulrmMjAzbs0nXyk1dDz/8sCRp9uzZdttdunRJCxYskMViUYcOHfJ3UgCAG2KkCQBwQ40aNVLZsmW1YMECpaenq2rVqjp58qRWrVqlixcvSpLt/3fddZeef/55RUVF6fHHH1fXrl1tU4ZfHW250W1tkuTh4aGJEydq9OjR6tmzp/r06SPDMLRs2TLbpBOO6mzevLlWrFihCxcuKCQkRNnZ2dqwYYPi4+PtZtS7+tzSwoUL1bJly3yFDA8PD33wwQc6efKk6tWrp927d2vNmjW67777NHDgQNt2jz32mGJiYjRo0CA9+eSTslqtWrVq1XWDUW7q6t69u7755hutXbtWp06dUvv27ZWWlqZVq1bpxIkTGjNmjKpUqZLn8wEAOEZoAgDckJ+fnz788EPNnDlTn332mTIzMxUQEKCHH35YAwYMUKdOnfTjjz/queeekyQNGzZM5cqV09KlSzVjxgz5+voqLCxMGRkZWrx48XWfd7pWly5d5Ovrq7lz52revHlyd3dXt27dVK1aNU2ePPmG+5lMJkVFRenDDz/Uhg0btG3bNklSUFCQZsyYYTfleb9+/fTrr79q1apVio6Ozldo8vb21rvvvqupU6dq3bp18vHx0VNPPaURI0bYPb/Vu3dvpaamavny5Zo2bZrKlSun7t27q1WrVnriiSfsjpmbuiwWi+bNm6ePPvpIa9eu1YwZM+Th4aF69epp/Pjx1/0gXQDArTMZt3J/AgAA/yc1NVXZ2dkqU6ZMjnXjx4/XZ599pm+//faGn0EEAEBRxTNNAIACcfjwYTVu3Fhz5syxW37x4kV99913Kl++/HWnygYAoKjj9jwAQIG4//77Vbt2bb3//vs6f/686tSpo6SkJK1evVrnzp3TO++84/CZJgAAiipuzwMAFJjz589r4cKF2rJli06fPi0PDw8FBwfrmWeeUdOmTZ1dHgAA+UJoAgAAAAAHeKYJAAAAABwgNAEAAACAA4QmAAAAAHDA6bPnZWVlae7cuVq7dq2SkpJUt25d/fOf/1SDBg0kSbGxsZoyZYr2798vPz8/Pf300+rfv/8ttWkYhqxWHuUCAAAA7mRmsylXM7s6PTT961//0sqVKzV16lRVqVJFCxYs0DPPPKOvv/5arq6uGjBggNq1a6dJkyZpz549mjRpkkqXLq2wsLB8t2m1Gjp//lIBngUAAACA4sbPr7QslmIQmrZs2aKuXbuqRYsWkqTIyEitXLlSe/bsUVxcnFxdXTV58mS5uLioRo0aOn78uObPn39LoQkAAAAAcsvpzzT5+/vru+++08mTJ5Wdna1PP/1Ubm5uCgoKUkxMjEJCQuTi8r9sFxoaqmPHjuns2bNOrBoAAADAncLpI01jx47VyJEj1b59e1ksFpnNZkVFRalq1ao6ffq0atWqZbd9hQoVJEmnTp1SuXLl8t2ui4vT8yIAAACAYsDpoenIkSMqU6aM5s6dq4CAAK1cuVJjxozR0qVLlZ6eLjc3N7vtS5UqJUnKyMjId5tms0m+vqVvqW4AAAAAdwanhqZTp05p9OjRWrJkiRo3bixJqlevno4cOaKoqCi5u7srMzPTbp+rYcnT0zPf7Vqthi5cSM1/4QAAAACKPW9vD1ksN78Dzamh6bffftPly5dVr149u+X169fXDz/8oEqVKikhIcFu3dWvAwICbqntrCzrLe0PAAAA3C5Wq1XZ2VnOLqNYs1hcZDYXzCM5Tg1NgYGBkqSDBw8qODjYtvzQoUOqXr266tevrxUrVig7O1sWi0WSFB0drbvvvlv+/v5OqRkAAAC4XQzD0IUL55WWluLsUkoEDw8veXv75eqzmBxxamgKDg7WAw88oIiICE2YMEGBgYFau3atduzYoeXLl6ty5cpauHChxo4dq2eeeUZ79+7VkiVLNGnSJGeWDQAAANwWVwOTl5ev3NxK3fIv+3cqwzCUmZmhlJRESZKPz60NuJgMwzAKorD8Sk5O1rvvvqtt27YpOTlZtWrV0qhRoxQSEiJJ2rt3r6ZMmaIDBw6ofPnyGjhwoPr163dLbWZnW/lwWwBAgTKbTTKb+eUmL6xWQ1arU38NAYoUqzVbCQkn5eXlKy8vb2eXUyKkpFxQSkqiKlSoct1b9a58uO3Nb+FzemhyBkITAKAgmc0mlS3rmas3XvxPdrZVSUmpBCfg/1y+nKlz507Jzy9Qbm6lnF1OiZCZmaHz50/L37+iXF3dcqzPbWhy+pTjAAAUd2azSRaLWXOXb1d8QrKzyykW7qrgo6GPPyiz2URoAv6GW/IKTkG9loQmAAAKSHxCso7FJzq7DAAlkLNuAeY22isITQAAAEAR5sxbgPN7G+3p06e1f/9v6tDh4dtUWeEiNAEAAABFmLNuAb6V22inTJmgwMCKhCYAAAAAhac43QJc0uaaIzQBAAAAKDDDhj2nPXt+1Z49v2r37l8kSW3atFd09HYlJp7XG29M06JFH6hixUoaO3ai3X7XLjt2LE5z5szSb7/tlqenpxo1aqJhw16Uv3+5Qj8n5kYFAAAAUGDefHO67r8/WO3aPaQFCz6WJK1e/ZlGjhyjd96J0n331bvpMc6e/a+GDn1GlStX1cKF/9bbb7+rS5dSNGTIQKWlpd3uU8iB0AQAAACgwHh7+8jFxUWlSpWSr6+vJCk09EE1adJUQUF15eaW8/OS/m7Nms9VvnyAXnxxjKpVq66goDqaPHmqzp8/p+++23K7TyEHbs8DAAAAcFtVrlwlT9sfOvS74uL+0EMPtbRbnpmZqWPH4gqytFwhNAEAAAC4rUqVKnXTbbKzs23/tloNNWrUWKNHR+bYzsurTIHWlhvcngcAAACgQJlMjj+I19XVVZcuXbJ9bbVa9ddfJ21f33NPDR0/fkwVKgSocuUqqly5iry9vTV79js6evTIbav7RghNAAAAAAqUh4enTp36SwkJZ667/v77g/XzzzsVHf3/dPLkn5o1a7ouXkyxre/Ro5dSUlI0efI4HT58SIcPH9Jrr72i2NgDuvvuGoV1GjbcngcAAAAUA3dV8Ck27T32WJimTJmgp556XB4eHjnW9+37pOLjT2r8+Ei5ubnqkUe6q0OHjrbPd6pU6S7NmfOB3n9/jl54YZAsFovq1auv2bPft00uUZhMRkn75KlcyM626vz5SzffEACAXHBxMcvXt7Refe/rYvPBk85W/S5fvTmyixITLykry+rscoAi4fLlTJ07d0r+/hXl6vq/GebMZpPKlvWUxVL4N4llZ1uVlJQqq7V4RoYbvaZX+fmVztXrykgTAAAAUIRZrYaSklJlNjt+Tuh2tV1cA1NBIjQBAACnccZfzos7fom9M/F9dy5CEwAAKHQ+ZdxlWK3y9s75rAMcs1qzlZiYxi/QQCEiNAEAgEJX2t1NJrNZcV8tUNq5U84up9jw8K+ou7s+K7PZRGgCChGhCcAtM5tNTrnPurjjVgtASjt3SmlnTji7DABwiNAE4JY4c0af4q64z0gEAMCdgtAE4JaYzSZZLGbNXb5d8QnJzi6n2Lirgo+GPv4gt9gAAFAMEJoAFIj4hGQ+nwYAAJRI3E8DAAAAAA4w0gQAAAAUcc6adIlJi64gNAEAAABFmNlskq+vh8xmS6G3XdQ/F6xXr27q3LmrBg0afFvbITQBAAAARdiVUSZLoX+uGZ8L9j+EJgAAAKAY4HPNnIfQBAAAAKDAtGjRWC+99LI2bvxaR44cUuXKVfTccy+oRYvWkqRFiz7Q7t2/yN/fXzt2/D917vyIXnrpZe3b95vef3+OYmMPqGzZsnrwwVYaMmSoSpf2kiSlpKTo3Xen66efvpeLi4v69Xu60M6J2fMAAAAAFKj335+jhx/uoiVLlqlZsxZ69dV/at++32zr9+z5VX5+5bR48Sfq1auvjhw5rBdffEFNmzbTRx8t14QJU3TwYKxeemmYDOPKrYGvvRap2Nj/6O23Z2nWrLnasWO7Tp8unNsVGWkCACeyWPjbVV4wixMAFA9dunRVWNg/JEnPPz9cu3f/os8//1T16tW3bTNo0GB5eV0ZRXr99fEKCQlV//4DJUlVqlTVxIlT9I9/dNfu3b+oXLly2rUrWu++O0/16zeUJE2Y8IZ69epWKOdDaAIAJ/Ap4y7DapW3t4ezSylWivosTgCAKxo1amz3db16wdq1K9r2ta+vny0wSdLBgwd18uQJPfRQyxzHOn78mJKTkyRJderUtS338/NXpUp3FXDl10doAgAnKO3uJpPZXOgzIRVnzOIEAMWHxWIfM7KzrXZTppcqVcpuvWFY1bFjZ9tI07XKlvVVTMxOScrx8//v7dwuhCYAcCJmQgIAlES//35ALVq0sn29f/9e1a4ddMPt7767huLijqpy5Sq2ZcePH9Pcue9pyJChqlmztiRp377f1Lx5C0nSxYsXFR//5206A3uEJgAAAAAF6rPPlqtq1eoKCqqjL75YoyNHDikycvwNt+/bt5+GDn1G77zztsLC/qGUlIt6552pysjIUJUq1eTq6qq2bTto1qxpcnV1lb+/v95/f64uX75cKOdDaAIAAACKAQ//isWmvcce66nPPlumo0ePqEaNmpo5c47uvbfmDbe///56mjlzjhYu/JcGDuwnT08PPfBAEw0d+qJcXV0lSePGTdScOe9pwoRXZbVa1b17TyUlJea7xrwgNAEAAABF2JWZQ7N1d9dnndB2dr6eI61e/R698MLI664bNGiwBg0anGP5Aw800QMPNLnhMUuVctfo0REaPToiz/XcKkITAAAAUIRZrYYSE9NkNpuc0jaT7xCaAAAAgCKP8OJchCYAAAAABeann2KcXUKB46PoAQAAAMABQhMAAAAAOEBoAgAAAIoQw+DZpYJSUK8loQkAAAAoAiwWiyQpMzPDyZWUHFdfS4vl1qZyYCIIAAAAoAgwmy3y8PBSSsqVD2x1cyslk6nwpxkvCQzDUGZmhlJSEuXh4SWz+dbGighNAAAAQBHh7e0nSbbghFvj4eFle01vRZEITWvXrtX8+fP1559/qmrVqho2bJg6d+4sSTp58qRef/11/fzzz/L09FSvXr00fPhw2/AlAAAAUFKYTCb5+PirTBlfZWdnObucYs1icbnlEaarnB6a1q1bp7Fjx+rVV19Vy5YttX79eo0aNUqBgYG6//77NWjQIFWvXl0rVqzQiRMnNHbsWJnNZo0YMcLZpQMAAAC3hdlsltns5uwy8H+cGpoMw9B7772n/v3768knn5QkPf/884qJidGuXbsUHx+vv/76S5999pl8fHxUq1YtnTt3TtOmTdOQIUPk5saFBAAAAOD2curseXFxcYqPj1e3bt3sli9atEiDBw9WTEyM7rvvPvn4+NjWhYaGKiUlRbGxsYVdLgAAAIA7kFNHmuLi4iRJqampGjRokA4cOKDKlSvr+eefV7t27XT69GkFBgba7VOhQgVJ0qlTp1S/fv18t+3iwmzrQEGwWOhLKFxF8ZorijWhZOOaAwqXU0NTSkqKJCkiIkLDhg3TmDFjtHHjRr3wwgtavHix0tPT5e3tbbdPqVKlJEkZGfmfv95sNsnXt3T+CwcAOI23t4ezSwCcjn4AFC6nhiZXV1dJ0qBBg9SjRw9JUp06dXTgwAEtXrxY7u7uyszMtNvnaljy9PTMd7tWq6ELF1LzvT+A/7FYzLx5o1BduJCm7Gyrs8uwQz9AYSuK/QAojry9PXI1cuvU0BQQECBJqlWrlt3ye++9V9u2bVNISIgOHTpkty4hIcFu3/zKyuIHDQAUR9nZVn6G445HPwAKl1NviL3vvvtUunRp/fbbb3bLDx06pKpVq6pJkyY6cOCA7TY+SYqOjlbp0qUVFBRU2OUCAAAAuAM5NTS5u7vrmWee0dy5c/XVV1/pxIkT+te//qXt27drwIAB6tChg8qXL68XX3xRv//+u7Zs2aKZM2dq4MCBTDcOAAAAoFA4/cNtX3jhBXl4eGjWrFk6c+aMatSooaioKDVt2lSStHDhQk2aNEn/+Mc/5OPjoyeeeEIvvPCCk6sGAAAAcKdwemiSpAEDBmjAgAHXXVetWjV9+OGHhVwRAAAAAFzBJP8AAAAA4AChCQAAAAAcIDQBAAAAgAOEJgAAAABwgNAEAAAAAA4QmgAAAADAAUITAAAAADhAaAIAAAAABwhNAAAAAOAAoQkAAAAAHCA0AQAAAIADhCYAAAAAcIDQBAAAAAAOEJoAAAAAwAFCEwAAAAA4QGgCAAAAAAcITQAAAADgAKEJAAAAABwgNAEAAACAA4QmAAAAAHCA0AQAAAAADhCaAAAAAMABQhMAAAAAOEBoAgAAAAAHCE0AAAAA4AChCQAAAAAcIDQBAAAAgAOEJgAAAABwgNAEAAAAAA4QmgAAAADAAUITAAAAADhAaAIAAAAABwhNAAAAAOAAoQkAAAAAHCA0AQAAAIADLnnd4ZVXXsn1tiaTSW+++WZemwAAAACAIiPPoen06dM6cOCAkpOTdddddykgIEBJSUk6fvy4DMNQYGCgbVuTyVSgxQIAAABAYctzaOrSpYsOHz6sZcuWqVGjRrblR48e1fPPP68nnnhCTz31VIEWCQAAAADOkudnmt5//32NGTPGLjBJ0j333KMXX3xRixYtKrDiAAAAAMDZ8hyazp8/Lx8fn+sfzGzWxYsXb7koAAAAACgq8hya6tevrzlz5igxMdFueUJCgqKiotSiRYsCKw4AAAAAnC3PzzRFRkaqX79+ateunRo2bChfX1+dO3dOu3fvlr+/v1599dXbUScAAAAAOEWeR5qCgoK0fv169e3bVykpKdq/f7/S09M1cOBArV69WhUrVrwddQIAAACAU+R5pEmSAgICFBERUdC1AAAAAECRk+eRJknKzMzUsmXLNGzYMPXp00d//PGHli9frr17995SMXFxcWrYsKFWr15tWxYbG6t+/fqpQYMGateunT7++ONbagMAAAAA8iJfs+eFhYVpypQpOn78uPbu3av09HRt27ZN4eHh2r17d74KuXz5ssaMGaPU1FTbssTERA0YMEBVq1bVqlWrNHToUM2YMUOrVq3KVxsAAAAAkFd5Dk3Tpk3TpUuX9PXXX2vNmjUyDEOSNHv2bNWrV0+zZ8/OVyFRUVHy8vKyW/bZZ5/J1dVVkydPVo0aNRQWFqann35a8+fPz1cbAAAAAJBXeQ5N3333nUaOHKlq1arJZDLZlpcqVUoDBw7Uf/7znzwX8fPPP+vTTz/V1KlT7ZbHxMQoJCRELi7/e/QqNDRUx44d09mzZ/PcDgAAAADkVZ4ngsjIyFDZsmWvu85isejy5ct5Ot6FCxf08ssva9y4cTlm3jt9+rRq1aplt6xChQqSpFOnTqlcuXJ5autaLi75epwLwN9YLPQlFK6ieM0VxZpQsnHNAYUrz6GpXr16WrZsmVq3bp1j3Zdffqn7778/T8ebOHGiGjZsqG7duuVYl56eLjc3N7tlpUqVknQlvOWX2WySr2/pfO8PAHAeb28PZ5cAOB39AChceQ5NI0eO1NNPP63u3burdevWMplM+uqrrxQVFaWffvpJCxcuzPWx1q5dq5iYGH355ZfXXe/u7q7MzEy7ZVfDkqenZ15Lt7FaDV24kHrzDQHclMVi5s0bherChTRlZ1udXYYd+gEKW1HsB0Bx5O3tkauR2zyHpsaNG2vx4sV65513tHDhQhmGoSVLlqhu3br64IMPFBoamutjrVq1SufOnVObNm3slk+YMEFff/21AgMDlZCQYLfu6tcBAQF5Ld1OVhY/aACgOMrOtvIzHHc8+gFQuPIcmnbs2KGGDRtqxYoVSk9PV3Jysry8vFS6dN5vd5sxY4bS09PtlnXs2FEjRozQo48+qnXr1mnFihXKzs6WxWKRJEVHR+vuu++Wv79/ntsDAAAAgLzK81OEw4cP16ZNmyRduX0uICAgX4FJujJaVK1aNbv/JMnf318BAQEKCwtTSkqKxo4dqyNHjmj16tVasmSJBg8enK/2AAAAACCv8hyavL295e7ufjtqycHf318LFy5UXFycevTooTlz5ujll19Wjx49CqV9AAAAAMjz7XmDBw/WG2+8obi4OAUFBV13QoYmTZrku6CDBw/afR0cHKxPP/0038cDAAAAgFuRq9CUkZFhm+p7woQJkqRZs2ZJkt0H3BqGIZPJpNjY2IKuEwAAAACcIlehqV27dpozZ44aNmyoJk2aqHfv3goMDLzdtQEAAACA0+UqNF28eNE21XdMTIz++c9/Kjg4+LYWBgAAAABFQa5CU7169TR69Gi9/fbbMgxDQ4cOlZub23W3NZlM2rJlS4EWCQAAAADOkqvQNHPmTC1ZskRJSUlas2aN6tatKz8/v9tdGwAAAAA4Xa5CU0BAgCIiIiRJO3fu1EsvvaSgoKDbWhgAAAAAFAV5nnJ869att6MO3GHMZpPMZtPNN4SN1WrIajWcXQYAAMAdJ8+hCbhVZrNJvr4eMpstzi6lWLFas5WYmEZwAgAAKGSEJhS6K6NMFsV9tUBp5045u5xiwcO/ou7u+qzMZhOhCQAAoJARmuA0aedOKe3MCWeXAQAAADhkdnYBAAAAAFCUEZoAAAAAwAFCEwAAAAA4QGgCAAAAAAcITQAAAADgAKEJAAAAABwgNAEAAACAA4QmAAAAAHCA0AQAAAAADhCaAAAAAMABQhMAAAAAOEBoAgAAAAAHCE0AAAAA4AChCQAAAAAcIDQBAAAAgAOEJgAAAABwgNAEAAAAAA4QmgAAAADAAUITAAAAADhAaAIAAAAABwhNAAAAAOAAoQkAAAAAHHBxdgHFndlsktlscnYZxYrFQlYHAABA8UFougVms0lly3oSAgAAAIASjNB0C8xmkywWs+Yu3674hGRnl1Ns1K9dSX06NXB2GQAAAECuEJoKQHxCso7FJzq7jGKjUnlvZ5cAAAAA5Br3lQEAAACAA4QmAAAAAHCA0AQAAAAADhCaAAAAAMABQhMAAAAAOEBoAgAAAAAHCE0AAAAA4AChCQAAAAAccHpoSkpK0muvvaZWrVqpUaNGevzxxxUTE2Nbv2PHDvXs2VP169dXp06dtH79eidWCwAAAOBO4/TQNGrUKO3evVszZ87UqlWrVKdOHQ0aNEhHjx7VH3/8ocGDB6tly5ZavXq1evfurZdfflk7duxwdtkAAAAA7hAuzmz8+PHj2r59u5YtW6YHHnhAkjR+/Hj9+OOP+vLLL3Xu3DnVrl1bL730kiSpRo0aOnDggBYuXKhmzZo5s3QAAAAAdwinjjT5+vpq/vz5qlevnm2ZyWSSyWTShQsXFBMTkyMchYaG6pdffpFhGIVdLgAAAIA7kFNDk7e3t1q3bi03Nzfbso0bN+r48eNq2bKlTp8+rcDAQLt9KlSooLS0NCUmJhZ2uQAAAADuQE69Pe/vfv31V73yyivq2LGj2rRpo/T0dLtAJcn2dWZm5i215eJy63nRYnH6I2G4wxTFa64o1oSSrShec0WxJpRsXHNA4SoyoWnLli0aM2aMGjVqpBkzZkiSSpUqlSMcXf3aw8Mj322ZzSb5+pbOf7GAk3h75/+6B0oK+gFAPwAKW5EITUuXLtWUKVPUqVMnvf3227bRpIoVKyohIcFu24SEBHl6eqpMmTL5bs9qNXThQuot1Sxd+SsPP7RQmC5cSFN2ttXZZdihH6Cw0Q+AotkPgOLI29sjVyO3Tg9Ny5Yt0+uvv67w8HCNHTtWJpPJtq5x48batWuX3fbR0dFq1KiRzOZbG5bOyuIHDYqf7Gwr1y7uePQDgH4AFDanhqa4uDi9+eabeuihhzR48GCdPXvWts7d3V3h4eHq0aOHZsyYoR49euj777/XN998o4ULFzqxagAAAAB3EqeGpo0bN+ry5cvavHmzNm/ebLeuR48emjp1qubNm6fp06fro48+UuXKlTV9+nQ+owkAAABAoXFqaBoyZIiGDBnicJtWrVqpVatWhVQRAAAAANhjvkoAAAAAcMDpE0EAAAAAdyqz2SSz2XTzDWHHajVktRqF1h6hCQAAAHCCK58d6iGz2eLsUoodqzVbiYlphRacCE0AAACAE1wZZbIo7qsFSjt3ytnlFBse/hV1d9dnZTabCE0AAAAoXrjVLG+ufqhq2rlTSjtzwsnVwBFCEwAAAG6Z2WxS2bKetiAAlCSEJgAAANwys9kki8Wsucu3Kz4h2dnlFAv1a1dSn04NnF0GcoHQBAAAgAITn5CsY/GJzi6jWKhU3tvZJSCXGD8FAAAAAAcITQAAAADgAKEJAAAAABwgNAEAAACAA4QmAAAAAHCA0AQAAAAADhCaAAAAAMABQhMAAAAAOEBoAgAAAAAHCE0AAAAA4AChCQAAAAAcIDQBAAAAgAOEJgAAAABwgNAEAAAAAA4QmgAAAADAAUITAAAAADhAaAIAAAAABwhNAAAAAOAAoQkAAAAAHCA0AQAAAIADhCYAAAAAcIDQBAAAAAAOEJoAAAAAwAFCEwAAAAA4QGgCAAAAAAcITQAAAADgAKEJAAAAABwgNAEAAACAA4QmAAAAAHCA0AQAAAAADhCaAAAAAMABQhMAAAAAOEBoAgAAAAAHCE0AAAAA4AChCQAAAAAcIDQBAAAAgAOEJgAAAABwoFiEJqvVqtmzZ6tly5Zq0KCBnn32Wf3555/OLgsAAADAHaBYhKZ58+Zp2bJlev3117VixQpZrVY988wzyszMdHZpAAAAAEq4Ih+aMjMz9eGHH2rEiBFq06aNgoKCNGvWLJ0+fVqbNm1ydnkAAAAASrgiH5p+//13Xbp0Sc2aNbMt8/b2Vt26dfXzzz87sTIAAAAAdwKTYRiGs4twZNOmTRo+fLh+++03ubu725aPHDlS6enp+uCDD/J8TMMwZLXe+mmbTJLZbFZySrqys623fLw7hZurRV6epXT50gUZ1mxnl1MsmMwWuZb2ltVqVVHrsfSD/KEf5B39oGShD+QP/aBkoR/kT0H2A7PZJJPJdNPtXG6tmdsvLS1NkuTm5ma3vFSpUkpOTs7XMU0mkyyWm784ueXj5X7zjZCDa2lvZ5dQ7JjNRXdwmH6QP/SDvKMflCz0gfyhH5Qs9IP8Kcx+UHR73P+5Orr090kfMjIy5OHh4YySAAAAANxBinxoqlixoiQpISHBbnlCQoICAgKcURIAAACAO0iRD01BQUHy8vLSzp07bcsuXLigAwcOqEmTJk6sDAAAAMCdoMg/0+Tm5qZ+/fppxowZ8vPz01133aXp06crMDBQHTt2dHZ5AAAAAEq4Ih+aJGnEiBHKysrSuHHjlJ6eriZNmmjRokVydXV1dmkAAAAASrgiP+U4AAAAADhTkX+mCQAAAACcidAEAAAAAA4QmgAAAADAAUITAAAAADhAaAIAAAAABwhNAAAAAOAAoQkAAAAAHCA04bYJDw+3/bt27dpOrARwrtmzZysmJqbAj7t8+XItX7483+tRdBXkNRMZGanVq1fnWL569WpFRkYWSBu4vu7duzu7BBQx/D5UfLk4uwCUXLt27XJ2CUCR8PPPP6tp06YFftzHH3/8ltaj6Lpd1wwK17p165xdAoACQmhCgXj//ff1xRdfyGKx6MEHH1RaWpokqWfPnra/cE6aNEm7d+9WRkaG3n77bQUHB+vEiROaOHGiEhMT5ebmpoiICDVq1EiRkZFKTEzUiRMnNHLkSHXq1MmZp4dibubMmdq4caMsFou6d++ujh076rXXXlNSUpI8PT01duxYBQcHKzIyUu7u7tqzZ4+SkpL00ksvacuWLYqNjVXbtm01duxYrV69Whs3blRKSooSEhLUunVrjR07VvHx8erfv7+2bt0q6cpf8Xft2qUmTZpo//79GjdunGbPnq3SpUvn+ZqfOnWq/Pz89Nxzz0mSIiIiFBISor/++kuSNHToUFv/slgsateunYYPH66oqChJ0vDhw/Xdd9/p3XffldVqVZUqVTR58mSVK1dO7dq1U/fu3bV9+3YlJSVp/PjxatmyZSF/h0q+nTt36l//+pfKlCmjP/74Q4GBgZo5c6a++uorrV27Vunp6ZKuXKv79u2zu2befPNNDRs2zBaiateurYMHDyoqKkp79uzR6dOn1bt3b9WtW1czZ85URkaGkpOTNXr0aD3yyCO5qm/Dhg1avHix0tPTlZ6ersmTJys0NFTh4eGqX7++YmJilJCQoOHDh6tHjx5KSUnRq6++qsOHD6t8+fIymUx64YUXJElz5szRv//9b0myuwaXLl2a41xr1aqlmJgYTZ48WWazWY0bN9b333+vzZs36/z583rttdds1/mwYcPUrl27HLWHh4fr7rvv1v79+5WWlqbIyEi1bt06x+vTunXr6/b7U6dO6ZVXXtHZs2fl5uamiRMnKjg4WOvWrdNHH32k7Oxs3XvvvZo0aZI8PT2v29f27NmjN954Q4ZhqFSpUnrjjTd0zz332H2vzpw5oxMnTig+Pt5WiyS99957Wr9+vcqUKaMaNWqoSpUqGj58eH4vNTjRzp07NXfuXLm4uOjYsWNq2bKlAgICtGXLFlmtVs2fP9+2bVpamsaNG6eDBw/KZDJp0KBBeuyxx274HpORkaGXX35ZJ06ckMlkUp8+fdS3b18nnu0dyABu0bZt24ywsDAjNTXVuHz5sjFkyBBj6dKlRq1atWzb1KpVy1i/fr1hGIbx0UcfGcOHDzcMwzD69u1r7N271zAMwzh+/LjRtm1b4/Lly0ZERIQxevTowj8ZlDgbN240+vTpY6Snpxvp6elGr169jCZNmhhff/21YRiGsXv3bqNNmzZGRkaGERERYQwZMsQwDMNYvXq18cADDxhnz541Ll68aDRs2NBITk42Vq1aZYSGhhoJCQlGRkaG0adPH+Prr782/vzzT6Nt27a2dletWmVEREQYhmEY/fr1M6Kjow3DyN81Hxsbazz66KOGYRhGenq60bx5c+PixYvG7NmzjdmzZxuxsbFGjx49bOtHjRplpKam2tafPXvWePDBB40TJ04YhmEYCxYssPXBtm3bGosWLTIMwzA2bdpkOw4KVnR0tNGgQQMjPj7eMAzDGDJkiLFkyRIjPDzcSE1NNQzDMN577z1j8uTJhmHYXzPX/tswDNvP1tmzZxuPP/64bfnw4cONQ4cOGYZhGDt27DC6du1qGIZhREREGKtWrcpR09VrNDs72wgPDzfOnj1rGIZhfP7558bgwYNtbV+t6T//+Y8REhJiGIZhTJ061XjjjTcMwzCMEydOGA0aNDCio6ON6Ohoo1+/frY2rl6DFy9evO65ZmZmGq1atbL1iQULFtj60ahRo4yNGzcahmEY586dMzp06GCr8Vr9+vUzXn75ZcNqtRoHDhwwQkNDjYyMjByvT1hY2HX7/eDBg40lS5YYhmEYO3fuNAYNGmQcOXLE6Nu3r5GWlmYYhmHMmzfPmDp16g372gsvvGB8++23hmEYxvr1643Vq1fn+F717NnTyMjIMFJSUowWLVoYv//+u7F161ajd+/eRlpampGammr07NnTmD17do5zRPFwbT9PTU01GjRoYCxfvtwwDMOIjIw0lixZYrsm3n77bWPSpEmGYVy5vtu1a2fExsbe8D1m8+bNxrBhwwzDMIzz588bY8aMcc5J3sEYacIti46OVteuXeXh4SFJCgsL09q1a3Ns17FjR0lSrVq1tHnzZl26dEn79u3TuHHjbNtkZWXp1KlTkqSGDRve/uJR4u3cuVOdO3dWqVKlJElLlixRmzZt1LlzZ0lSgwYN5OPjo6NHj0qS2rRpI0mqVKmSatasKX9/f0lS2bJldeHCBUlS27ZtVb58eUlSly5d9PPPP6tevXo3rSW/13xQUJAk6Y8//tChQ4cUGhoqLy8v2/qqVasqMzNTTz75pFq3bq2XXnrJ1h8lae/evQoODlaVKlUkSX369LH7i2fr1q1t7SQlJd30PJA/NWvWVKVKlSRJderU0cWLFzVr1ix9/fXXOnbsmH788UfVqVMnT8ds0KCB7d/Tp0/Xd999p02bNum3337TpUuXcnUMs9msefPmaevWrYqLi9OuXbtkNv/vkeer10edOnVs18dPP/2k6dOnS5KqVKmi5s2bO2zDy8vruud66NAh+fn52fpPnz59tGzZMlsbhw8f1ty5cyVd6St//PGHrU9eq3fv3jKZTKpTp44CAwN18OBBu9fn0qVLOn78+HX7/c6dO23nEhISopCQEC1dulTHjx9Xnz59bG1XqVLlhn2tXbt2GjdunNq2bau2bdvq4YcfzlFjs2bN5ObmJjc3N1WrVk3Jycnavn27unbtKnd3d0nSo48+avs5g+Kpdu3atn7u6+urZs2aSbrynnLt9zY6OlpTpkyRJPn5+al9+/batWuXvLy8rvseM3jwYE2ZMkWDBg1S69atFRERUchnBkITbpnVas2xLCsrK8cyF5crl5vJZLLt5+bmZnfP95kzZ2w/KK79pQ/IL4vFYrvmJCk5OTnHNoZh2K5ZV1dX2/Kr1+zfXbvcarXKbDbLZDLJMAzb8suXL+fYL7fX/LXB6v7779eUKVPUvXt3bdiwQQcPHlSvXr3sjuvp6am1a9dq586d+umnn9S3b1/b7VFX2/37+V5b39VAee3rhIJ39XWWrrzWf/31l3r37q3w8HC1atVK5cqVU2xs7HX3vXptZWZm2i2/9ufkE088oZCQEIWGhqpZs2YaM2aM3bbLly/XihUrJEl9+/a11XPp0iWFhYXp0UcfVZMmTVS7dm198sknOeq+9vqwWCx21/u15/X3fuDq6qpTp07pySefzHGuFovluu8h0pXr9uOPP1bZsmUlSQkJCfLz89Ozzz6rhIQESbKFf4vFYrff1a+vvj7Xq/Vqv3dxcbE7t8OHDys7O1tdunSx9cPU1FRlZmbesK+FhYWpWbNm2rZtm5YsWaJt27bpjTfesGvv799/wzBkNptveP4onq59D5Hsr81r/f2avPZ96HrvMQEBAdqwYYO2b9+uH3/8UT169ND69evl7e1dwGeAG2H2PNyy0NBQffXVV0pLS1NWVpZWrVqlJk2ayGKxXDc8XVWmTBlVr17d9gtkTEyMevbs6XAfIK9CQkK0efNmZWZmKjMzU0OGDNGlS5e0YcMGSdKePXuUkJCgWrVq5fqYP/74oy5cuKCMjAytX79eLVq0kI+Pj5KSkpSQkKDs7Gxt2rTJtr3FYlF2dnaur/l69epp3bp1Wrdune0vkd26ddPGjRsVGxurFi1a2G0fExOjZ599VqGhoYqIiFCNGjUUFxdnW1+/fn3t3btXf/75pyTp008/VUhISB5eRdwO+/btU/Xq1TVgwADVr19fP/zwg7KzsyX975qRrvy1+vfff5ckffPNN9c9VlJSko4dO6YXX3xRrVu31vbt2237X/X444/brqtrJwk5duyYTCaTnn/+eYWGhtrVcSMPPvig1qxZI0k6ffq0du7cKZPJJF9fXx07dkxpaWlKS0vTtm3bHJ7rPffco4sXL+o///mPJNndpRAaGmobdTp27Ji6du2q5ORkLViwwHYeAQEBkqT169fb2klKSsrRn728vFSlSpXr9vuQkBDb/rt379aoUaPUtGlTbd68WWfPnpUkvfXWW5o3b94N+9qzzz6ruLg4PfHEExo5cqQOHDjg8PW79nXcsGGDMjIylJmZqQ0bNvDHiztEaGioVq5cKUk6f/68tmzZosaNG0u6/nvMl19+qYkTJ6p9+/YaN26cPD09bXcpoHAw0oRb1rZtW8XGxqpXr17KyspS8+bN1b9/f+3du1ePPvqoPv/88xvuO336dE2cOFELFy6UxWLRe++9Jzc3t0KsHiVdhw4ddODAAYWFhclqtSosLEytW7fWxIkTNW/ePLm6uioqKipP11358uU1ePBgnT9/Xl27drXd0jdkyBD17dtX5cqVU+PGjXX+/HlJV275mzBhgt566618X/Ply5dXQECAatSokeMvlw888IDuuece220+devWVatWrWy/iJYrV06TJ0/WsGHDlJWVpcDAQL355pu5Pl/cHi1atNDvv/+uLl26yM3NTcHBwTp06JAk+2tm8ODBioiI0Jo1a9SsWTPbyOS1ypYtq969e+uRRx6Rl5eX6tevr/T09FzdohcUFKS6deuqc+fOcnd3V5MmTRQfH3/d0Zmrnn/+eY0fP17dunVT+fLlValSJbm7u6tmzZrq2LGjunbtqoCAAD3wwAOSroSD5cuX5zhXNzc3zZw5U+PHj5dhGAoKCrLdqjZu3DhNmDBB3bp1k2EYmjJlynVvzZOujNj26NFDVqtVM2fOvO4o8dW+9/d+P378eI0bN07Lli2Tm5ub3n77bQUFBWnYsGEaMGCArFaratSoocjISHl6el63r/n7+2vSpEmaMWOGXFxccj2Ve+vWrbVv3z716NFDpUuXlq+vr92IFEquoUOHauLEieratauys7P13HPPKTg4WEeOHLnue0xGRoa+/fZbPfLII3J1ddXDDz/M9OWFzGQ4+qkIALBzdVa8qVOnOrsUwGm++OILBQYGKiQkRCkpKerZs6dWrlwpHx+fPB3HMAxNmzZNQ4cOlZeXl7Zs2aIvvvhCs2fPzvUxwsPD7WYXLE5+++03HTp0SL1795ZhGBoxYoTCwsJsf4jBnYf3mKKLkSYAAJAn99xzjyZMmGC7jW/kyJF5DkzSlWd7/P399Y9//EOurq7y9/fX66+/XtDlFlnVq1fX3Llz9fHHH0u6MsJIYAKKJkaaAAAAAMABJoIAAAAAAAcITQAAAADgAKEJAAAAABwgNAEAShwe1wUAFCRCEwCgRPn2228VERHh7DIAACUIU44DAEqUJUuWOLsEAEAJw0gTAAAAADjA5zQBAEqM8PBw7dq1y/b1xx9/LB8fH82ZM0cxMTG6ePGi/Pz89PDDD2vMmDFyd3eXJKWkpGjatGnavHmz0tPT1aZNG9WvX19vvfWWDh486KzTAQAUEYQmAECJceTIEf3zn/+UJE2YMEHly5fXo48+qgYNGig8PFxubm764YcftHjxYo0ePVrPPfecJKl///6KjY3VSy+9pEqVKmnZsmXasWOHMjMzCU0AAJ5pAgCUHPfee6+8vLwkSQ0aNNBPP/2kOnXq6L333rMtb968ubZv366dO3fqueee044dO7Rz505FRUWpY8eOkqRWrVqpa9eu+uOPP5x2LgCAooPQBAAosVq0aKEWLVro8uXLOnLkiI4fP65Dhw7p/PnzKlu2rCQpOjparq6u6tChg20/s9msLl26KCoqykmVAwCKEkITAKDEslqtmjlzpj755BOlpqaqYsWKCg4OVqlSpWzbJCYmqmzZsjKb7edG8vf3L+xyAQBFFKEJAFBizZ8/X0uWLNGkSZPUsWNHlSlTRpLUq1cv2zYBAQFKTEyU1Wq1C07nzp0r9HoBAEUTU44DAEqUa4PPL7/8onvvvVdhYWG2wHTmzBkdOnRIVqtVkhQSEqKsrCxt3brVtp9hGNqyZUvhFg4AKLIYaQIAlCje3t7avXu3duzYoWrVqumnn37S/Pnz1aBBAx0/flwffPCBMjMzlZaWJklq0qSJHnzwQY0dO1Znz55VpUqV9Pnnn+vgwYMymUxOPhsAQFHAlOMAgBIlOjpar7zyiv773//q9ddf1759+7Rp0yZdvHhRFStW1COPPCKTyaQPPvhA27dvl7e3t5KTkzV16lRt2bJFWVlZat++vby9vbV27Vr9+uuvzj4lAICTEZoAAHe0+Ph47dmzR+3bt7d92K0kjRgxQn/++afWrFnjxOoAAEUBt+cBAO5oZrNZkZGRat++vXr16iWLxaIff/xRmzZt0ltvveXs8gAARQAjTQCAO150dLTmzp2r2NhYZWVlqUaNGhowYIC6du3q7NIAAEUAoQkAAAAAHGDKcQAAAABwgNAEAAAAAA4QmgAAAADAAUITAAAAADhAaAIAAAAABwhNAAAAAOAAoQkAAAAAHCA0AQAAAIADhCYAAAAAcOD/A9Es/zlA6gXfAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Few-shot with GPT 4\n",
    "method = \"few_shot\"\n",
    "model = \"gpt-4-0613\"\n",
    "y_pred[method][model], performance[method][model] = evaluate(\n",
    "    test_df=test_df, model=model, system_content=system_content,\n",
    "    assistant_content=assistant_content, tags=tags)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "61521970",
   "metadata": {},
   "source": [
    "As we can see, few shot learning performs better than it's respective zero shot counter part. GPT 4 has had considerable improvements in reducing hallucinations but for our supervised task this comes at an expense of high precision but lower recall and f1 scores. When GPT 4 is not confident, it would rather predict `other`."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e04b53bf",
   "metadata": {},
   "source": [
    "## OSS LLMs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c6c72add",
   "metadata": {},
   "source": [
    "So far, we've only been using closed-source models from OpenAI. While these are *currently* the gold-standard, there are many open-source models that are rapidly catching up ([Falcon 40B](https://huggingface.co/tiiuae/falcon-40b), [Llama 2](https://ai.meta.com/llama/), etc.). Before we see how these models perform on our task, let's first consider a few reasons why we should care about open-source models.\n",
    "\n",
    "- **data ownership**: you can serve your models and pass data to your models, without having to share it with a third-party API endpoint.\n",
    "- **fine-tune**: with access to our model's weights, we can actually fine-tune them, as opposed to experimenting with fickle prompting strategies.\n",
    "- **optimization**: we have full freedom to optimize our deployed models for inference (ex. quantization, pruning, etc.) to reduce costs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15ea136e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coming soon in August!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3724d63b-58f8-4374-a89d-275a83c8190e",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec0b498a-97c1-488c-a6b9-dc63a8a9df4d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"zero_shot\": {\n",
      "    \"gpt-3.5-turbo-0613\": {\n",
      "      \"precision\": 0.7919133278407181,\n",
      "      \"recall\": 0.806282722513089,\n",
      "      \"f1\": 0.7807530967691199\n",
      "    },\n",
      "    \"gpt-4-0613\": {\n",
      "      \"precision\": 0.9314722577069027,\n",
      "      \"recall\": 0.9267015706806283,\n",
      "      \"f1\": 0.9271956481845013\n",
      "    }\n",
      "  },\n",
      "  \"few_shot\": {\n",
      "    \"gpt-3.5-turbo-0613\": {\n",
      "      \"precision\": 0.8435247936255214,\n",
      "      \"recall\": 0.8586387434554974,\n",
      "      \"f1\": 0.8447984162323493\n",
      "    },\n",
      "    \"gpt-4-0613\": {\n",
      "      \"precision\": 0.9407759040163695,\n",
      "      \"recall\": 0.9267015706806283,\n",
      "      \"f1\": 0.9302632275594479\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(performance, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cc80311",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transform data into a new dictionary with four keys\n",
    "by_model_and_context = {}\n",
    "for context_type, models_data in performance.items():\n",
    "    for model, metrics in models_data.items():\n",
    "        key = f\"{model}_{context_type}\"\n",
    "        by_model_and_context[key] = metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6771b1d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA9gAAAGACAYAAABWaMrCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABmJ0lEQVR4nO3dd3gU1f/28Xt304EQAkLoHUJVpCsgIL1IR1RAUJp0pCs/eu+9CYiAgtKlRLGAIlJUUERAFOktICWU1N15/sjDflmTQAiTyvt1XVxkZ87OfmZ2z87eO7NnLIZhGAIAAAAAAE/EmtwFAAAAAACQFhCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATuCV3AQCAlO/kyZP65JNP9MMPP+jy5ctyc3NT4cKF9corr6h169Zyc/vf7qRmzZq6cOGCy/09PDwUEBCgevXqqWfPnvL09NSQIUO0cePGhz5uhQoVtHLlyhjT9+/fr/bt27tMc3d3V7Zs2VS3bl316tVL3t7eT7DGCTNnzhzNnTtXf/75Z5I/9sNs2LBBQ4cO1TfffKNcuXIldzkAAKRZBGwAwENt375dQ4cOVcGCBdWxY0flz59fYWFh+u677zR+/Hjt3r1b8+fPl8Vicd7npZdeUvfu3Z23w8PDtX//fs2fP18XLlzQ9OnT1b17d7Vp08bZZv78+Tp69Kjmzp3rnJY+ffqH1jZ8+HCVKFFCkhQaGqrjx49r9uzZunr1qqZMmWLWJgAAAIgXAjYAIE4nT57U0KFDVbVqVc2cOdPlSPVLL72kihUrqnfv3goKClKDBg2c8/z9/fXcc8+5LKtixYq6fPmyNmzYoCFDhihPnjzKkyePy308PDxi3O9hChUq5NK+cuXKun37thYsWKARI0Y8MqADAACYid9gAwDitGTJElmtVo0aNcolXN9Xt25dNW3aNN7LK1mypAzD0KVLl0ys0pWvr2+MaTdv3tTw4cP1wgsvqFSpUmrdurX27t3r0qZo0aL6+OOP9f7776tChQoqU6aM+vTpo2vXrrm027Rpk5o1a6Znn31W1atX17Rp0xQREeHSZteuXXrllVdUqlQp1a1bV5s2bXLO279/v4oWLaq9e/eqXbt2Kl26tKpXr661a9cqODhYPXv2VJkyZfTSSy9p+fLlLss9fvy4evbsqUqVKqlEiRKqWrWqxo4dq7CwMJf1mDt3rpo3b67SpUu7nBFwX0hIiJo0aaKaNWvq4sWLkqQ9e/aodevWKlOmjMqXL6933nlHJ0+ejNc2BwAA0QjYAIA4ffPNN6pUqZIyZ84cZ5tJkya5HL1+mFOnTkmScufObUp9DodDUVFRioqKUmhoqA4ePKgVK1aoadOmzqPX4eHhevPNN/XNN9+oX79+mjt3rgICAtSpU6cYIXvGjBlyOByaPn26Bg0apJ07d2r8+PHO+R9//LEGDx6sEiVKaO7cuerSpYtWrlypsWPHuixn+PDh6tChgxYsWKCAgAANGTJEx48fd2nz7rvvqmbNmlq0aJHy58+vESNGqH379ipcuLDmz5+v0qVLa8KECTp8+LAkKTg4WG+88YZCQ0M1ceJEffDBB2rYsKFWrlypFStWuCx74cKFaty4sWbPnq26deu6zLt79646d+6skJAQrVixQjly5NC5c+fUvXt3lSxZUgsWLNC4ceN06tQpdenSRQ6H48meJAAAniKcIg4AiNWtW7d069Yt5cuXL8a8qKgol9sWi0U2m8152zAMlzb//vuvvv/+e61Zs0YNGjSQv7+/KTV26NAhxrRcuXKpb9++ztubN2/W8ePH9dlnn+nZZ5+VJFWrVk3t2rXT1KlTtX79emfbIkWKaMKECc7bhw8f1hdffCEpOszPmzdPtWrVcgnUoaGh2rZtmyIjI53Txo4dq2rVqkmS8uTJo9q1a+vAgQMKDAx0tmnRooU6duwoSfLx8VHr1q1VunRp9enTR5IUGBioHTt26ODBgypdurROnDihYsWKadasWc4vD1544QXt2bNH+/fvV5cuXZzLLleunHPZkvT7779Liv6y4Z133tGVK1e0cuVK54Bnhw8fVlhYmLp27aps2bJJkgICAvTNN9/o3r17nGoPAEA8EbABALGK68jlmTNnVKdOHZdpOXPm1Lfffuu8vWnTJpfToiXJzc1NtWvX1ogRI0yrcdSoUc5BziIiInTu3DktXrxYLVu21KeffqocOXJo7969euaZZ1SiRAmX0F+jRg1NnjxZt27dUsaMGSUpxu+/AwICFBoaKin66Pu///6r2rVru7R5++239fbbb7tMK1eunPPv+yE2JCTEpU2ZMmWcf98/Q+D+FwCSlClTJknS7du3JUlVqlRRlSpVFBkZqb///ltnzpzRiRMndP36dfn5+bksu1ixYrFur0GDBunIkSMaP368y1kEzz77rDw9PdWyZUvVq1dP1apVU8WKFVW6dOlYlwMAAGJHwAYAxCpTpkzy8fGJccmt7Nmza926dc7b8+bN04kTJ1za1KhRQz169JAUfXTb29tbOXPmlJeXl6k15s+fX6VKlXLeLlu2rCpUqKBatWpp2bJlGjZsmG7evKmrV686g/h/Xb161Rmw/3tpL6vVKsMwJEX/jlvSQ0+Xv8/Hx8dlGZKcy7kvtqPCD7u02P1T1z/++GPdu3dP2bNnV+nSpeXp6fnQx3/QlStXVKJECc2bN0/16tVTunTpJEV/CbBq1SotXrxY69at04oVK+Tr66vXX39dffv2dRkhHgAAxI2ADQCIU82aNbVz507duXPHGQg9PDxcQu1/j57en/Zgm6SUI0cO+fv76/Tp05KkDBkyKF++fJo6dWqs7eN7Xej7g6ddv37dZfqNGzd09OhRlyPSiWHx4sVavny5Ro0apTp16ihDhgySpJYtW8Z7GXPnzpW3t7eaN2+uGTNmaNiwYc559wdEi4iI0C+//KJPP/1UCxcuVGBgoOrXr2/6+gAAkBYxyBkAIE5dunRRVFSUhg0bFmOkbEkKCwvTuXPnkqGyuJ0/f17Xr193/na8QoUKunTpkjJnzqxSpUo5/+3Zs0dLlixx+e34wxQoUECZMmXSzp07XaZv3rxZXbp0cfkNdmL45ZdfVKhQIbVo0cIZrq9cuaITJ07EeyCyLFmyqGjRourQoYM+/vhj/fbbb5Kk5cuXq0aNGoqIiJCHh4cqV66sMWPGSJJzlHEAAPBoHMEGAMSpaNGimjJlioYOHarmzZurZcuWKlq0qKKionTo0CGtW7dO165dU6dOnZKlvr///tt5irRhGLp48aLmzZsnT09PtW3bVpLUvHlzrVq1Sh07dlS3bt2UPXt2/fjjj/rggw/Utm1bubu7x+uxbDabevXqpdGjRytz5syqWbOmTp06pdmzZ+uNN95wnmaeWEqXLq358+dr8eLFeu6553TmzBktWrRIERERzt+Jx1fPnj0VFBSkYcOGacOGDapUqZKmTp2qHj16qG3btrLZbFqzZo08PDxUo0aNRFojAADSHgI2AOCh6tatq5IlS2r16tVat26dLly4IMMwlDt3bjVo0EBt2rSJdaTxpDB69Gjn31arVX5+fnruuec0ZcoUZ00+Pj76+OOPNW3aNE2ZMkW3b99Wzpw51b9/f7311luP9XhvvPGGfHx8tHTpUn366acKCAhQ586d1blzZzNXK1Zdu3bVjRs3tGLFCs2bN0/Zs2dXkyZNZLFYtGjRIoWEhMR6DfDYeHt7a/jw4eratasWL16sHj16aOHChZo3b57effdd2e12lSxZUsuWLVOBAgUSec0AAEg7LMZ/R10BAAAAAACPjd9gAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgArfkLiAlMAxDDgeXA08OVquFbQ/Egr4BxES/AGKiXyQPq9Uii8WS3GUgBSJgS3I4DF2/fje5y3jquLlZlSlTOoWE3FNUlCO5ywFSDPoGEBP9AoiJfpF8/P3TyWYjYCMmThEHAAAAAMAEBGwAAAAAAExAwAYAAECa98cfR9Sx4+uqVauKunfvpAsXzsdoc+/eXY0fP0oNGrysFi0aaePGdTHaGIahnj27aOnSRTHmXbx4QfXqVU+M8gGkEgRsAAAApGnh4eF6770Bev319goK2qny5Stq+PChMdrNmTNTly5d1Jo1GzRnziKtXr1S33+/y6XN2rWrdfjwrzHue+TI7+rVq6vu3LmTSGsBIDUgYAMAACBNO3jwZ/n6+qp27Xpyd3dX+/Zv6cKF8zp16h+Xdrt371Lnzu/I1zejcuTIqaZNWygoaKtz/tmzZ/T55xtVrVp1l/sdOLBPI0e+pzfeeDPxVwZAisYo4gBStT/+OKKpU8fr3LmzKlIkUO+/P1I5c+ZyaXPv3l3NnDlVP/zwvby9vdW2bQc1a9ZSkhQcfEWTJ4/TkSOH5eXlrebNW6l9+7dc7n/x4gW99dYb+uKLXUm1WgAAE509e1p58+Zz3rbZbMqZM5fOnj2t/PkLOKc7HA55eXk5b1utVl28GH0qud1u1/jxo9S370B9+eV2l+UXLRqo1as36Nq1q4m7IskkMfe14eFhmjhxrPbu/UE+PunUufM7ql+/UZKvY2Ky2+2KjIxM7jKQQO7u7rLZbPFuT8AGkGrdP+WvZ8++ql79Za1atVzDhw/V0qUrXdo9eMrfnTt31Ldvd2XOnEXVqlXXuHGjVLRoUU2cOF3Xr/+rLl06qHjxkipXroKk6FP+RowYmqBT/hL6gaRVq9aSoj+QTJo0XocP/yYvLy81adJcHTp00uXLl9WuXasY26JhwyYaPPj9RK/vwQ9M06ZNTLT6AMAsoaGh8vT0cpnm5eWlsLAwl2kvvFBFS5cu0rBho3X7doi2bv1cUVFRkqTVq1eqYMFCKleuQoyAnTGj3xPXmJD35PbtO6pTpw6SEu89ObH3tYsWzVNYWKg2bfpCp0//o/79e6lQoSIqXLjIk23QFMAwDF26dEk3b96UwaXKUy2LRfLz81P27Nnjde1zAjaAVOvBU/4kqX37t/Tpp5/o1Kl/XI5I7N69S+PHT5Gvb0b5+mZ0nvJXrVp1TZkyU1arVW5ubrp165YcDofSp08vKfqUv8mTx+mNN97U9OmTHqu2J/lAkjXrM2ratJHGjh2lPHnyaezYybp27ar69u2uHDlyqU6devrqq93OZRw/flRDhvTXm2++nST13f/ANGHCaOXNmz9R6gMAM3l5eSk8PNxlWlhYmLy9fVym9e7dX9OmTdRrrzVTjhy5VL9+Q+3c+Y3++eektm37PMZ7pFkS+p7cr18P5cuXS2XLVk609+TE3td+9dWXmjJlpry8vBQYWFy1atXVV18FpYmAfenSJd24cVMZMvjJ09NTEtfNTn0MhYeH68aNm5KkHDlyPPIeBGwAqZYZp/x5eHhIkjp3bq9jx46qYcNXFBhYXNKTnfL3JB9Itm/fqldeaSAPDw+1b99R7u7uyp49h6pUeUl//HFYderUc94/KipK48aNVPfuvRUQEJAk9QUFbVWVKtUStT4AMFPevPkUFLTNedtut+vChXPKkyevS7sbN65r4MD3nOFv0aJ5KlSosHbv3qVr166qefOGkqLDudVq1Z9/HtPkyTOfuL6Evic3a9ZCGzduVJkyFRPtPTkx97UhISG6ceO68uT53/Lz5MmrAwf2xXfTpVh2u103b0aH6wwZMiZ3OXgCHh7Rr+ubN28qW7ZsjzxdnEHOAKRaj3vK3+3bt3Xx4gVt3fq5wsMjXNrMnfuBVq/eoIMHf9GmTeslRZ/y5+7unqDaHvaB5EGxfSA5f/6crFarpk6dKX//zJKiPxT99NM+FShQyOX+27Z9Lh+fdKpTp36S1Xfx4nlZrVZNmjQj0eoDADM9/3w53bhxXUFBWxUZGakVK5YpR45cypcvv0u7FSuWavHiebLb7Tp27A9t2bJRjRo11Ztvvq2vvtqtL77YpS++2KXatevpjTfeNCVcS0/ynmzT2bNnE/U9OTH3tWFhoc7l3efp6aWwMNezDVKjyMhIGYb+/5FrpHaenp4yDMXrt/QEbACp1uOc8ufl5a3XXmumkSPfV/36DZ1HJ+7z9PRU7tx51Lx5K/344w9PXNuTfCCJiHD9QGK32zVu3Ei5u3u4DPxiGIbWrFmldu06JGl9//3AlBj1AYCZPD29NHnyTK1f/5kaNHhZP/20X2PGTJQktW3bWjt2BEmSunfvo/Pnz6l+/ZoaNWqY+vYdqJIlSyV6fQl9T96yZVOi7zMSc197P1g/uPzw8DD5+Hg/Vo0pG6eFpw3xfx45RRxAqvWkp/wZhqGOHd/QsGGjVKhQYUlSZGSEMmRw/UCQEE/ye79du75xzg8NDdXw4UN0/fp1TZs2x3manSQdPfqH7t69q8qVqyRpfTt3Jn59AGC2wMBiWrJkRYzpq1Z95vw7c+Ysmj597iOX9f77I2Odnj17Dv3ww8+PXVtC35MbNGik77771jk/Md6TE3Nf6+ubUX5+mXTu3BkVKRIoKfpSaLlzuy47rbFYLLJakz54OxyGDEZbS3QcwQaQaj3pKX8Wi0UFCxbSsmWLFR4eplOn/tHGjetUp06DJ64tb958OnfurPP2oz6QbN36tRYvXq47d+6oUKHogV1CQkLUs2cXWSxWzZ27WH5+fi733bdvj6pUqfZYl44wp77CiV4fADxNEv6efFuBgdHBNLHekxN7X1urVh0tXbpI9+7d1fHjx/TVV1+qdu26j1VjamKxWJQhg5d8fb2T/F+GDF7xGgU7Jfjll59VqdLzunjx4iPbXrx4UZUqPa9ffnn8L7cSA0ewAaRa90/5mzp1gqZPn6zChYu4nPLXvn1H1alTX92799G4cSNVv35N+fv7u5zy17fvQE2fPknNmjVUhgwZ1KlTN1WsWPmJa3vwA0mtWnW1atXyOD+QpEuXXn36DNCJE8e1ZctGTZ06U5I0bNgQZcsWoDFjJsb6gejo0T9Us2atJK9v4sQZkqQRI4YmWn0AkFBWa/IcHYwvh8OQw+F6FDGh78mbN2/UokWLJCXee3Ji72u7du2pGTMmq1WrV+Tp6aVevfqpcOGij11namG1WmSzWTX14190/srtJHvcXNkyaMAbZWW1WmS3p/yj2KVLP6tt23bIzy/TI9tmy5ZN27btkK9vyhhMzmJwnoDsdoeuX7+b3GU8ddzcrMqUKZ1u3LirqChHcpcDmO748WOaOnWCzpw5rcKFizivafrgB5J//72mceNG6siR3+Xv769OnbqpXr36unr1gho3biwPD0/ZbP872ahOnfoaOPA9SVLbtq3Us2c/Var0QpLWV6tWXf3zz99q375NotYHPIh9BuLDarXIz8/H5X0ppbHbHbp5816MkJ2Q9+SuXburdevm+vnn3/TGG615T05C/v7pHvo6CwsL08mT/yhLlgB5ePxvoDObzSpfX2/1nb5LJy/cSopSJUkFc2bUzHerKyQkVHY776GPKyIiXNeuXVbBggVcBuWLDQFbBOzkwoclIHb0DSAm+gXi4/7rJKmPDsbX/aOIZr2O6RfJ52kK2JUqPa8BAwYrKGib/vrrhHLnzqOuXXuoWrWXJEkffLBQBw/+rMyZs+jHH/eoQYNGGjBgsA4f/k3z58/WsWNH5eeXSVWqVFX37r2ULl307/SjoiK1bNkSbd++RTdu3FT+/Pn1zju9VLFiJf3yy8/q0aOLNmzYqhw5cuiPP45o9uzpOnHiT7m5uals2fLq27e/AgKy6+LFi2revJHmzVussmXLyW6367PPVmvjxvW6fPmSAgKyq02bN9S8eUtJ0aef9+79jqZMmaG5c2fp3LmzypEjp3r06K1q1arHug0eJ2BzijiAVCE1nvIHAEge56/cTtLwkpak5P0t+9rkM3/+HHXv3kvDh4/W1q2fa8iQ/lq4cKlKl35WknTo0EG9+uprWrlytex2h/7664R69XpHHTu+rffeG6Hr1//VnDkz1Lt3dy1Z8pEsFoumT5+qnTu/1sCBQ1WkSFFt2bJZAwf21cqVa1we2263a8CAPmrSpLlGjBijkJAQTZo0TmPHjtLcuQtj1Dp79nQFBW1T//6DVaxYCe3du0czZkxRRES42rR5w7nMuXNn6d13Bypr1gAtWDBHo0YN15YtX8jHxyfGMh8HARtAipdaTvm7fTvMlNE5769nSl5fM/GBCcDTxqz3d7P3FxaLRRl8PWWzpszBKe0Ou27eCGWfkQwaNGisli1flST16NFbBw/+rLVr1zgDtiR17txN6dNnkCSNHDlMFStWUocOb0uS8uTJozFjJqh588Y6ePAXBQYW05Ytm9S//yDn2ADvvNNTkqG7d13PLL57965u3rypLFmeUUBAduXIkVNjx07UjRvXY9R59+4drV+/Vn36vKu6des7H/vSpQv66KMP9eqrrzvbdu3aXeXKVZAkvfVWZ+3c+Y1OnvxLpUo9G2O5j4OADSDFS64BQeKrWH5/dWlSUn5+T/aN53/5+ppzHVCHwyGrNeWGdT4wAXha+GXwlOFwmPb+fp/Zy5u9b5kuhFw2dZlPKqdvgHpXektWq4X9RTIoW7acy+1SpZ7VgQP7nLczZfJ3hmtJ+vPP4zp37qxq1HgxxrJOnz4lHx8fRUZGqkQJ1+vMv/NOL0lyGRHc19dXbdu+qWnTJmnx4gUqX76CKld+UbVq1Y5l2acVFRWlZ58t4zK9TJmyWrPmE12//r9Q/uAggvdPW4+MjIp7I8QTARtAqpFST/nLlTW9LFargjfNVMS/55O7HBc+BcrIv8YbKfLDksQHJgBPl/Te7il2fyH9b59xIeSyTt04l9zlIAVxc3ONjf/98t7T0zPG/Lp16zuPYD8oU6ZMunTp0mM9fo8evdWiRSv9+OMe/fTTfk2bNkmrVn2kFStWu7SL60xCh8MRYz3c3T1itDPjTEQCNgCYJOLf84q4fCq5y3DhnjmnJPFhCQBSkJS4v5D+t88A/uvYsaOqWvUl5+3ff/9NRYsGxtm+YMFCOnXqlHLnzuOcdvr0Kc2ZM1Pdu/dS7ty55ebmpmPHjqpw4SLONm+/3V61atVRkSL/W/aZM6e1Zs0n6tu3v5o3b6nmzVvqt99+Vdeub+mvv04oUyZ/Z9v8+fPLzc1Nv/12SEWK/O9yb7/9dkiZM2eRr6/vE2+LRyFgAwAAAADitGbNJ8qbN5+KFSuuTZs26K+/Tui994bH2f7119uqa9dOmjJlglq2fFV37tzWlCkTFR4erjx58srd3V2tWrXRokXz5eeXSQUKFNCWLZt08uTfGj58tK5du+Zclp+fn7766guFh4epXbsOstls2rZti3x9fZUvXz7duhXibJsuXXo1bdpCH3ywUBkz+qlYseLav3+v1q9fq27despiSfwB/AjYAAAAAJCEcmXL8OhGKejxmjVroTVrPtbJk3+rUKEimjVrvsuR5/8qWbK0Zs2aq0WLFqhDhzfk7e2tcuUqqHfvfnJ3d5ckde/eSzabTZMnj9Pt23dUuHBhTZ8+R3nz5nMJ2Bkz+mnGjDmaP3+OOnXqILs9SiVLltbs2QuULl16l4AtSX379pefn5/mzZut69f/Ve7cedS//2A1bdr8ibZBfBGwAQAAACAJOByG7HaHBrxRNskf2253JHi8k/z5C6hXr76xzuvcuZs6d+4WY3q5chWco3THxt3dXT179lHPnn1izCtbtpz27TvovF2q1LNasGBJrMvJkSOHS1s3Nzd16tRVnTp1jbX9f5cd2zKeBAEbAAAAAJKAYRi6fTssWa417nAYpgzihYcjYAMAAABAEjEMQ3Y7QTetImADAAAAAGJl1qnTTwvro5sAAAAAAIBHIWADAIAk9ccfR9Sx4+uqVauKunfvpAsXzsdoExUVpUmTxqlRo1pq1Ki2Zs6cJofDIUn655+/Va1aBdWuXdX5b9eubyRJZ8+eUe/e3VSnzkvq2PF1/forR14AAEmHgA0AAJJMeHi43ntvgF5/vb2CgnaqfPmKGj58aIx2Gzas1ZUrl7R27RatXPmZ9u/fq82bN0uS/v77L73wQhV99dVu57/q1V+W3W7Xe+8NUJEigdq27Wv16NFXQ4b019WrwUm9mgCApxQBGwAAJJmDB3+Wr6+vateuJ3d3d7Vv/5YuXDivU6f+cWl3/vzZ/39JGbskyWq1ytPTU5L0998nVKhQzOuvnj17RpcuXVS3bj3l7u6ucuUqqFSp0tq585vEXzEAAETABgAASejs2dPKmzef87bNZlPOnLl09uxpl3aNGzfVyZN/qX79mmrcuLby5cuvBg0aSIo+gv3bb4fUokUjtWzZWCtXfihJcjgccnd3l81mcy7HarXq4sWYp6ADAJAYCNgAACDJhIaGytPTy2Wal5eXwsLCXKZFRESqTp162rr1a61d+7lOnfpHH3/8sSTJ1zejXnihqlatWqspU2bp8883afv2LcqbN58yZvTTRx8tVWRkpA4e/FkHD/6s8PCIJFs/AMDTjYANAEAak5BBxObMme4cROw+wzDUs2cXLV26KMb9HzbvYby8vBQeHu4yLSwsTN7ePi7TJkwYpdq168nX11fZs+dQhw5va926dZKkkSPH6bXX2srb21v58xdQ8+at9MMP38vNzU0TJkzVgQP71KRJPW3evF4vv1xH6dOnj3d9ibntHjY4G4Cnh8Vikc1mTfJ/FosluVf9qUDABgAgDUn4IGL79OWX213arF27WocP/xrr4zxs3sPkzZtP586ddd622+26cOGc8uTJ69IuODhYkZGRzttubm5yc3NTWFiY5s2bpTt37jjnRUZGyMPDQw6HQxEREZo/f4m2b/9Go0ZN0OnTp1SoUOF41ZbY2y6uwdkAPD0sFosy+nrK19c7yf9l9PVMNSH7l19+VqVKz+vixYuSpHfe6azRo0ckc1Xx45bcBQAAAPM8OIiYJLVv/5Y+/fQTnTr1j/LnL+BsF3MQMYs8PDyc88+ePaPPP9+oatWqx3iMh817lOefL6cbN64rKGiratWqq1WrlitHjlzKly+/S7uKFStryZKFGj9+qsLDw7Ry5XI1atRQXl5eOnBgrxwOh955p5fOnj2tDRvWavDgYbJYLHrvvYHq0aOPqld/WTt2BOnChfOqWvWleNWW2NsursHZADw9rFaLLFabgjfNVMS/STc+hEfmXMratK+sVovsdiPJHvdpRMAGACANedggYg+GxMaNm6pfv56qX7+mHA6HatSopZdfriMp+qjy+PGj1LfvwBhHZh82Lz48Pb00efJMTZ06QdOnT1bhwkU0ZsxESVLbtq3Vvn1H1alTXwMHDtXMmVPVpk1T2WxuatiwsTp06KDbt8M1duxkTZs2UQ0avKz06dOrXbuOqlz5RUnSqFHjNXXqBE2cOFYFCxbS9Olz5OOTLkVsu7///kuRkZFq0aKRLBaLmjRprnbtOj72NgSQ+kX8e14Rl08ldxlIBARsAADSkMcdRKxDh866e/eOBg3qq/XrP1OLFq21evVKFSxYSOXKVYgREh82L74CA4tpyZIVMaavWvWZ829f34waPnyM87abm1Vubm6SwpU7dx7NnDk/1mWXKvWsPvpoTYLqSuxt5+ubUUWLFlPTpi10+fIlDRrUT5kzZ1GDBo0TVC8AJIVKlZ7X22931rZtWxQZGaUFC5Yoe/bsWrRovr78crvu3LmjAgUKqkuXd1SxYmXn/Y4e/UPz58/RH3/8Li8vb1WvXlN9+vSTl5e3QkJCNHfuLO3d+4OuX78hX98Mqlq1ut59d4C8vLyTcW2fHL/BBgAgDUnoIGLt27+lbds2659/Tmrbts/Vo0efGMt+2Ly0IDG3nRT34GwAkNKtX79WEyZM1aRJU5UnTx6NGTNCBw7s08iR4/TRR6v18st11L9/H+3Zs1uSdPHiBfXo0UXPPPOMliz5SBMnRg9AOXly9BlLY8aM0IkTxzVhwlStXbtJffr0V1DQVm3atCE5V9MUHMEGACANyZs3n4KCtjlvP84gYjabm3bv3qVr166qefOGkqIDptVq1Z9/HlOJEqXinDd58swYtVitFlmt5gyoY7NZXf43g8NhyOH4328RE3PbjRkzUUuWLNKbb77tHNX8/uBsAJDS1avXUMWKFZcknTt3Vjt2fKEVK1arSJGikqTXX2+rv/8+oVWrVujFF6tq06YNypgxo95/f8T/P/tIeu+9/9Phw79JkipUqKgyZco6B6HMkSOH1q5do5Mn/06GtTMXARsAgDQkoYOIffzxCtWsWUuvv95eb775trPduHEjFRCQXW+/3VWSHjrvQVarRZn8vGW12UxdP19f804ddNjtunEz1BmyE3vbxTU4GwCkdLlz53H+feLEn5Kkrl3fcmkTFRWl9OkzSJJOnvxLRYsWc4ZrSSpbtrzKli0vSWrRorV27/5O27Zt0blzZ3Xq1D+6ePGC8uZ1fb9NjQjYAACkIQkdRKx+/UZq3fp10+qwWi2y2pJ+pNz4enBE3fsBO7G33cMGZwOAlMzT09P5t8PhkCQtXLhUPj6uP6Gx/f8vVd3c3ONclsPhUP/+ffTPPydVp0491apVR0WLBmrixLGJUHnSI2ADAJDGJGQQsbi8//7IBM27L7WNlJuY2+5hg7MBQGpRsGAhSdK//15T0aJVnNMXLJgrm82mLl3eUb58+fXll0Gy2+3O0L1r17eaOXOaRo8er71792jJko9UsmQpSVJUVKTOnz+vnDlzJf0KmYxBzgAAAAAA8VKgQEG9+GJVTZo0Xrt3f6cLF85r5crlWrHiQ2dAbtnyVYWE3NKkSeN16tQ/OnToF82dO1Ply1dQ9uw5ZLO56ZtvvtLFixd07NhRvf/+EP377zVFREQk89o9OY5gAwCQSpk5iJjZzByMLDGl1Dr/OwAbgLTFI3PSHqk1+/HGjZuohQvnadKkcQoJCVHOnLn0/vvD1bBh9GUHn3nmGc2aNU9z587Sm2++Ll9fX9WqVUfduvWUl5eXhg8fpQ8+WKj16z+Tv39mValSVW3avKEffvjO1DqTAwEbpvjjjyOaOnW8zp07qyJFAvX++yNjnOIRFRWladMmaffunZIsqlevgUaMiB7cJTw8TBMnjtXevT/IxyedOnd+R/XrN5IUPYrrokVztX37FhmGoTp1GqhXr36yWlPmhyIASApWq0V+fj4pNiCmdLZ0fnIYDlMHTTOT3WHXzRuhhGwgjXE4DBkOu7I27Zvkj2047Al6T9m372CMaV5e3urbd4D69h0Q5/1KlXpWixYti3Ve3br1Vbdu/RjT+/btL0kqW7acy+MuWPDB45adbAjYeGLh4eF6770B6tmzr6pXf1mrVi3X8OFDtXTpSpd2Gzas1ZUrl7R27RaFhYWpd++u2rx5s6pXr6NFi+YpLCxUmzZ9odOn/1H//r1UqFARFS5cRGvWrNKhQwe1atU6GYahvn3f0RdfbFODBo2TaY0BIPlZrRbZbFZN/fgXnb9yO7nLieH5wKxq36B4cpcRJ6tXOlktVs3et0wXQi4ndzkucvoGqHelt1wGYAOQNhiGoVsh4cly9pHDYcgweE9JbARsPLGDB3+Wr6+vateuJ0lq3/4tffrpJzp16h/lz1/A2e78+bOy2x1yOOySJKvV6hyR8KuvvtSUKTPl5eWlwMDiqlWrrr76KkiFCxfRli2bNGTI/8nPz0+SNGnSTLm5mXvZFwBIrc5fua2TF24ldxkx5MqaPrlLiJcLIZd16sa55C4DwFPEMAzZ7QTdtIrzyvDEzp49rbx58zlv22w25cyZS2fPnnZp17hxU508+Zfq16+pxo1rK1++/GrQoIFCQkJ048Z15cnzv2XkyZNXZ86c1r1793T+/DmdP39Wbdo0V7NmDbRt22ZlzpwlaVYOAAAAAOKJgI0nFhoaKk9PL5dpXl5eCgsLc5kWERGpOnXqaevWr7V27ec6deofffzxxwoLC3Xe5z5PTy+FhYXrzp3o0x6/+26nFi/+UPPnL9HXX3+poKCtibxWAAAAAPB4CNipyB9/HFHHjq+rVq0q6t69ky5cOB+jTdu2rVW7dlXnv+rVK6lNm+aSpHv37mr8+FFq0OBltWjRSBs3rnPeLzj4igYP7qf69WuqWbMGWr58Sbzr8vLyUnh4uMu0sLAweXu7Xnh+woRRql27nnx9fZU9ew516PC21q1b5wznDy4jPDxMPj7ecnd3///r1UG+vhmVPXsONWnSXHv2fB/v+gAAAAAgKRCwU4n7A4m9/np7BQXtVPnyFTV8+NAY7Vat+kxffbVbX321Wxs3Bilr1mzq1aufJGnOnJm6dOmi1qzZoDlzFmn16pX6/vtdkqQJE0Yre/ac+vzzLzV//hIFBW3Vjh1fxKu2vHnz6dy5s87bdrtdFy6cU548eV3aBQcHKzIy0nnbzc1Nbm5uypgxo/z8MuncuTPOeWfPnlHu3Hnl55dJ6dNn0J07d5zzHA6HGJ8BAAAAQEpDwE4lHhxIzN3dXe3bv6ULF87r1Kl/4rzPggWzVbZseb34YlVJ0u7du9S58zvy9c2oHDlyqmnTFgoK2iqHwyEPDw+1b99R7u7uyp49h6pUeUl//HE4XrU9/3w53bhxXUFBWxUZGakVK5YpR45cypcvv0u7ihUra8mShbpz547+/feaVq5crrp160qSatWqo6VLF+nevbs6fvyYvvrqS9WuXVcWi0V169bX6tUrFRISosuXL2nTpvWqXr1mgrYjAAAAACQWAnYqEd+BxO47ffqUvv76S3Xt2tM5zeFwuPzO2Wq16uLF87JarZo0aYb8/TNLir5e9U8/7VOBAoXiVZunp5cmT56p9es/U4MGL+unn/ZrzJiJkqJPWd+xI0iSNHDgUGXJ8ozatGmqt95qq/LlK6pDhw6SpK5deypjRj+1avWK3ntvgHr16qfChYtKknr06KtChYqoXbvW6tSpnerWbaA6dWJeN+9hnvT0+qioKE2aNE6NGtVSo0a1NWfOdDkcDpf7X7x4QfXqVX+sugAAAACkHcl+mS6Hw6G5c+dq7dq1un37tsqXL6/hw4crd+7csbb/999/NX78eO3Zs0eGYeiFF17QkCFDlC1btiSuPGnFdyCx+z799BM1atTEeWkrSXrhhSpaunSRhg0brdu3Q7R16+eKiopyuZ/dbte4cSPl7u6h+vUbxbu+wMBiWrJkRYzpq1Z95vzb1zejhg8f47zt5maVm5ubpHB5e3vrvfdGxLpsDw+PR17I/mHie53uB2u9c+eO3nrrDefp9f+9hnevXl1VqNB25zY6cuR3jRgx1OVUdgAAAABPl2Q/gj1//nx98sknGjNmjNasWSOHw6FOnTopIiIi1vZ9+/bVxYsX9eGHH+rDDz/UxYsX1aNHjySuOunFdyAxSYqMjNS33+5Qw4ZNXKb37t1fXl7eeu21Zho58n3Vr99Q6dP/7zqloaGhGjLkXZ09e0bTps2Rh4dH4qxMEjPj9PqY1/C2OLfPgQP7NHLke3rjjTcTf2UAAACQqlksFtls1iT/Z7FYElTvsWNH9eqrzVW1akXNnj3DOf23337VCy+UM2uzpBnJegQ7IiJCy5Yt04ABA1S9enVJ0owZM1S1alXt2LFDjRq5HkENCQnRgQMHtGDBAhUrVkyS1KVLF3Xv3l03b950OVqb1uTNm09BQduct+MaSEySfv/9N/n7Z1aBAgVdpt+4cV0DB77nDNWLFs1ToUKFJUVv2379eihz5iyaO3exvL29Y63DarXIak1Y5/wvm83q8r8ZHA5DDofrCGgPO70+f/4CMZZx//T6Tz/d7JzWuHFT9evXU/Xr15TD4VCNGrX08st1JElFiwZq9eoNunbtqmnrAQAAgLTHYrEog6+nbFZbkj+23WHX7ZBwGY85WvDy5Uvl7u6u1avXKUOGDJKiw/XAgf1i/GQSyRywjx8/rrt376py5crOab6+vipevLh++umnGAHby8tL6dKl06ZNm1ShQgVJ0ubNm5U/f375+vomae1J7cGBxGrVqqtVq5bHOpCYJB09ekQlSpSKMX3FiqVKly69+vQZoBMnjmvLlo2aODH6W6gRI4YqW7YAjRkzUTZb7B3earUok5+3rHHMTyhf39jDfEI47HbduBnqErLNOL3+/jW8O3TorLt372jQoL5av/4ztWjRWhkz+sW6HAAAAOBBVqtFNqtNs/ct04WQy0n2uDl9A9S70luyWi2y2x8vYN++fVtFihRVrly5FRUVpZkzp2nduk9VsGAhhYTcSqSKU69kDdiXL0e/qLJnz+4yPWvWrM55D/Lw8NDEiRM1fPhwlStXThaLRVmzZtWqVatktT7ZUVA3t2Q/W/6h3Nx8NH36LE2aNF4zZkxW4cJFNH78JLm5WfXaay315ptvqV69BpKir2n9zDPPxFin3r37afTo4apfv6b8/f3Vv/9gPffcszp58m/99NN+eXp6ugzSVa9eAw0e/L7zts1mldVmU/CmmYr4N+YgYcnNI3MuZW3aV+7uNtnt//s2zcfHR5GRES7bIzw8TOnTp4uxjaJPr/9KH3zwocu8CRNGafjw0fL395O/v586duykVatW6NVX2zjb3D8Sn9JfS6mRmWc5IGXiOX58bLO0j+f48bHN0r608hxfCLmsUzfOJXcZj9S0aUNdvnxJkrR9+1atWLFav/56ULNmzdOlS5c0duzI5C0wBUrWgB0aGipJMX7r6+npqVu3Yn4bYhiGjh07pjJlyqhTp06y2+2aMWOGunfvrtWrV7v8nvhxWK0WZcqULkH3TUqVK5fXpk0bY0z/4osgl9sTJoyN9f6ZMqXTihUfxZhertyz+vPPP+NdR8S/5xVx+VS82ye1/x4RL1kyUF9+uc35HEefXn9epUoVi/G879u3T888k0Vly5Z2mX71arC8vGzO9hkzppOXl4fL/e/ejX7c1PBaAlIaM89kAdIK+gUQE/0iaX344SoNGtRPWbNm07vvDpSfXyYtX/6xJGnr1s+TubqUKVkD9v1LRkVERLhcPio8PDzW3wAHBQVp1apV2rlzpzNML1y4UDVq1NC6deucl3x6XA6HoZCQewm679PEZrOmije1kJBQlyPYRYqU1NWr1/Txx5+qdu26WrHiQ+XIkVP+/gG6ceOuy3337/9FxYuXjDG9UqUXNG3adE2aNE3h4eFauHCRXn65tku7W7eivzD6733x5FLLaw8J999+i0ejX6R99IvHR79I+1JKv/D19U4zR9MfJlOmTHJzc5enp5cyZ86S3OWkCskasO+fGh4cHKw8efI4pwcHB6to0aIx2v/888/Knz+/y5HqjBkzKn/+/Dpz5swT1RIVlfwd9T4zBxJ7GtntDpfn083NQ5Mnz9TUqRM0ZcpEFS5cRKNHT1RUlENt27ZW+/YdndfVvnjxojJlyhzj9dC//xDNnDlVLVs2kc3mpvr1G6lly9dc2t1/s09JryUgtfhvvwVAvwBiQ79ASpesATswMFDp06fX/v37nQE7JCRER48eVdu2bWO0DwgI0LZt2xQeHi5PT09J0r1793T+/Hm98sorSVp7YrFaLfLz83kqvhFLSvG5TrckDRgwJNb7//ca3rHJnj2Hfvjh54QXCQAAACBVS9aA7eHhobZt22rq1Kny9/dXzpw5NWXKFAUEBKhOnTqy2+26fv26MmTIIC8vLzVt2lRLly5V37591adPH0nSzJkz5enpqebNmyfnqpjGao2+Lt7Uj3/R+Su3k7scF88HZlX7BsWTuwwAAAAASJGSNWBLUu/evRUVFaVhw4YpLCxM5cuX19Kl0ddaO3/+vF5++WVNmDBBzZs3V9asWfXJJ59oypQpevPNN2W1WlWuXDl98sknzmuypRXnr9zWyQspa9j7XFkTNohcUkvJR/9ju043AAAAgLQh2QO2zWbTwIEDNXDgwBjzcuXKFWN064IFC2rhwoVJVR5SEVs6PzkMR4oe3MTusOvmjVBCNgAAwFMsp29Amn68p1myB2zALFavdLJarJq9b5kuhMS8jnpyy+kboN6V3pLVaiFgAwAAPIUcDkN2h129K72V5I9td9gT9Bl0wYIPYp3eqNEratQobYyDZSYCNtKcCyGXderGueQuAwAAAHBhGIZuh4QnyxWDHA5DhsFBnsRGwAYAAACAJGIYhux2gm5alXJHgwIAAAAAIBUhYAMAAAAAYAICNgAAAAAAJiBgAwAAAECi4LfWaUP8n0cCNgAAAACYyN3dXRaLFB4entylwATh4eGyWKKf10dhFHEAAAAAMJHNZpOfn59u3LgpSfL09JSU9JfmwpMyFB4ertu3bypTJj/ZbLZH3oOADQAAAAAmy549uyTp5s2bun07mYtBglksUqZMfs7n81EI2AAAAABgMovFohw5cihbtmyKjIxM7nKQQO7u7vE6cn0fARsAAAAAEonNZnusgIbUjUHOAAAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAE7gl9I4RERFat26dfvzxR129elXjx4/XgQMHVKJECZUuXdrMGgEAAAAASPESdAT7+vXratGihcaNG6czZ87o8OHDCgsL065du9SuXTsdOnTI7DoBAAAAAEjREhSwJ0+erLt372r79u3auHGjDMOQJM2ePVulSpXS7NmzTS0SAAAAAICULkEBe+fOnerTp4/y5s0ri8XinO7p6am33npLf/zxh2kFAgAAAACQGiQoYIeHh8vPzy/WeTabTZGRkU9SEwAAAAAAqU6CAnapUqX0ySefxDpvy5YtKlmyZLyX5XA4NHv2bFWtWlXPPfecOnfurHPnzsXZPjIyUtOmTXO2b9u2rY4dO/bY6wAAAAAAgJkSFLD79OmjPXv2qEmTJpo1a5YsFou2bt2qbt266YsvvlCPHj3ivaz58+frk08+0ZgxY7RmzRo5HA516tRJERERsbYfOXKkNmzYoPHjx2v9+vXy9/dX586ddfv27YSsCgAAAAAApkhQwC5Xrpw+/PBDeXt7a8mSJTIMQ8uXL9fVq1e1aNEiVapUKV7LiYiI0LJly9S7d29Vr15dgYGBmjFjhi5fvqwdO3bEaH/u3DmtX79e48aNU9WqVVWwYEGNHTtWHh4eOnLkSEJWBQAAAAAAUyT4Otjly5fXmjVrFBYWplu3bil9+vRKly7dYy3j+PHjunv3ripXruyc5uvrq+LFi+unn35So0aNXNrv2bNHGTJkULVq1Vzaf/vttwldDQAAAAAATJHggL148WL9/PPPWrx4sby8vLR//371799f3bp1U9u2beO1jMuXL0uSsmfP7jI9a9asznkPOnXqlHLnzq0dO3Zo8eLFunLliooXL64hQ4aoYMGCCV0VSZKbW4IO5pvOZksZdSDx8Bw/PrZZ2sdz/PjYZmkfz/HjY5ulfTzHSOkSFLCXLVummTNnugTpPHnyqF69epo4caI8PT3VqlWrRy4nNDRUkuTh4eEy3dPTU7du3YrR/s6dOzpz5ozmz5+vQYMGydfXVwsWLNDrr7+u7du3K3PmzAlZHVmtFmXK9HhH34GE8vX1Tu4SgBSHfgHERL8AYqJfIKVLUMBes2aN+vbtqy5dujinZc+eXcOGDVOWLFm0fPnyeAVsLy8vSdG/xb7/txR9GTBv75idx83NTXfu3NGMGTOcR6xnzJihl156SRs3blSnTp0SsjpyOAyFhNxL0H3NZrNZeeNI40JCQmW3O5K7jFSFfpH20S8eH/0i7aNfPD76RdqXUvqFr683R9MRqwQF7CtXrqhUqVKxznv22We1YMGCeC3n/qnhwcHBypMnj3N6cHCwihYtGqN9QECA3NzcXE4H9/LyUu7cuXX+/PnHWYUYoqKSv6Pi6WC3O3i9Af9BvwBiol8AMdEvkNIl6GuXnDlzau/evbHO++mnnxQQEBCv5QQGBip9+vTav3+/c1pISIiOHj2q8uXLx2hfvnx5RUVF6ffff3dOCwsL07lz55Q3b97HXAsAAAAAAMyToCPYrVu31pQpUxQZGalatWopc+bMun79unbu3KkPP/xQ/fv3j9dyPDw81LZtW02dOlX+/v7KmTOnpkyZooCAANWpU0d2u13Xr19XhgwZ5OXlpXLlyumFF17Q4MGDNXr0aPn5+Wn27Nmy2Wxq0qRJQlYFAAAAAABTJChgd+jQQVeuXNHKlSu1fPly53SbzaY333xTHTt2jPeyevfuraioKA0bNkxhYWEqX768li5dKnd3d50/f14vv/yyJkyYoObNm0uS5syZo6lTp6pnz54KCwvT888/rxUrVsjf3z8hqwIAAAAAgCkSfJmuwYMHq3v37jp06JBu3bolX19flS5dWpkyZXqs5dhsNg0cOFADBw6MMS9Xrlz6888/XaalT59eI0eO1MiRIxNaOgAAAAAApktwwJakDBkyqFq1ambVAgAAAABAqpWggB0WFqYFCxZo586dCg0NlcPhOpKfxWLR119/bUqBAAAAAACkBgkK2OPGjdO6detUoUIFFStWTFYr14ADAAAAADzdEhSwd+zYoX79+qlLly5m1wMAAAAAQKqUoEPPkZGRKl26tNm1AAAAAACQaiUoYFepUkXff/+92bUAAAAAAJBqJegU8QYNGmjEiBG6fv26nn32WXl7e8do07Rp0yetDQAAAACAVCNBAbtv376SpE2bNmnTpk0x5lssFgI2AAAAAOCpkqCA/c0335hdBwAAAAAAqVqCAnbOnDkfOt8wjAQVAwAAAABAapWggC1J27dv14EDBxQREeEM1IZh6N69e/r1118ZBA0AAAAA8FRJUMCeO3eu5s6dqwwZMigqKkru7u5yc3PT9evXZbVa1apVK7PrBAAAAAAgRUvQZbo2btyopk2b6sCBA+rQoYNq1KihH3/8UevWrZOfn58KFy5sdp0AAAAAAKRoCQrYV65cUePGjWWxWFSsWDEdOnRIklSyZEl169ZNa9euNbVIAAAAAABSugQFbB8fH1ksFklS3rx5df78eYWFhUmSihUrpvPnz5tXIQAAAAAAqUCCAnapUqWc17/Onz+/bDab9u7dK0k6efKkPDw8TCsQAAAAAIDUIEGDnHXr1k0dO3ZUSEiIFi5cqFdeeUWDBw9WxYoV9cMPP6hWrVpm1wkAAAAAQIqWoIBdvnx5rVu3Tn/++ackafjw4bJarTp48KDq1aunIUOGmFokAAAAAAApXYKvgx0YGKjAwEBJkqenp8aMGWNaUQAAAAAApDYJDthXrlzRkSNHdPv27VjnN23aNKGLBgAAAAAg1UlQwN6+fbuGDBmiiIiIWOdbLBYCNgAAAADgqZKggD1z5kyVLl1aQ4cOlZ+fn8klAQAAAACQ+iQoYAcHB2v06NEqUaKE2fUAAAAAAJAqJeg62M8995yOHz9udi0AAAAAAKRaCTqCPWLECHXr1k137txRqVKl5OPjE6NN+fLln7g4AAAAAABSiwQF7NOnT+vatWuaO3eupOhBze4zDEMWi0XHjh0zp0IAAAAAAFKBBAXsSZMmKU+ePOrcubOyZMlidk0AAAAAAKQ6CQrYFy9e1MKFC/XCCy+YXQ8AAAAAAKlSggY5K1KkiC5dumR2LQAAAAAApFoJOoI9dOhQDRgwQHa7Xc8995zSp08fo02OHDmeuDgAAAAAAFKLBAXsjh07KioqSsOHD3cZ4OxBDHIGAAAAAHiaJChgjxo1yuw6AAAAAABI1RIUsC9duqS6deuqYMGCZtcDAAAAAECqlKBBzhYtWqTz58+bXQsAAAAAAKlWggJ2oUKFdOrUKbNrAQAAAAAg1UrQKeI1atTQ9OnTtXv3bhUtWlQ+Pj4u8y0Wi3r06GFKgQAAAAAApAYJCthz586VJO3Zs0d79uyJMZ+ADQAAAAB42iQoYB8/ftzsOgAAAAAASNUSFLAfdPLkSd2+fVv+/v7KkyePGTUBAAAAAJDqJDhgb926VZMmTdK1a9ec07JkyaL+/furadOmZtQGAAAAAECqkaCA/e2332rgwIGqVKmS3n33XWXJkkXBwcH6/PPPNXToUPn5+al69eomlwoAAAAAQMqVoIC9YMEC1atXTzNmzHCZ3qJFC/Xr10+LFi0iYAMAAAAAnioJug72iRMn1KxZs1jnNWvWjEHQAAAAAABPnQQF7EyZMunWrVuxzrt586Y8PDyeqCgAAAAAAFKbBAXsypUra+7cubp8+bLL9EuXLmnevHl68cUXTSkOAAAAAIDUIkG/wX733XfVokUL1alTR2XKlFGWLFl07do1HTp0SBkzZlT//v3NrhMAAAAAgBQt3keww8PDnX8/88wz2rhxo9q1a6fQ0FAdOXJEoaGhateunTZu3KicOXMmSrEAAAAAAKRU8T6CXbNmTc2dO1dlypTR3Llz1apVKw0cODAxawMAAAAAINWI9xHs27dvKzg4WJI0b948XblyJdGKAgAAAAAgtYn3EexSpUqpf//+mjRpkgzDUI8ePeIcLdxisejrr782rUgAAAAAAFK6eAfs6dOna/ny5bp586Y2btyo4sWLy9/fPzFrAwAAAAAg1Yh3wM6WLZsGDx4sSfrqq6/Ur18/BQYGJlphAAAAAACkJgm6DraXl5f++ecfs2sBAAAAACDVSlDAjoyMVKZMmUwpwOFwaPbs2apataqee+45de7cWefOnYvXfT///HMVLVpU58+fN6UWAAAAAAASKkEBu3379po5c6YOHTqk0NDQJypg/vz5+uSTTzRmzBitWbNGDodDnTp1UkRExEPvd+HCBY0ePfqJHhsAAAAAALPE+zfYD9q8ebMuXryo119/Pdb5FotFR48efeRyIiIitGzZMg0YMEDVq1eXJM2YMUNVq1bVjh071KhRo1jv53A4NHDgQJUoUUL79u1LyCoAAAAAAGCqBAXsV155xZQHP378uO7evavKlSs7p/n6+qp48eL66aef4gzYCxcuVGRkpHr27EnABgAAAACkCAkK2D179jTlwS9fvixJyp49u8v0rFmzOuf91+HDh7Vs2TKtW7dOV65cMaUOSXJzS9DZ8qaz2VJGHUg8PMePj22W9vEcPz62WdrHc/z42GZpH88xUroEBez7vvvuO/3444+6evWq+vXrp2PHjqlEiRLKmTNnvO5///fbHh4eLtM9PT1169atGO3v3bunAQMGaMCAAcqXL59pAdtqtShTpnSmLAt4FF9f7+QuAUhx6BdATPQLICb6BVK6BAXs0NBQ9ejRQz/++KPSp0+vu3fv6u2339bq1at19OhRrVq1SoULF37kcry8vCRF/xb7/t+SFB4eLm/vmJ1n7Nixyp8/v9q0aZOQsuPkcBgKCbln6jITymaz8saRxoWEhMpudyR3GakK/SLto188PvpF2ke/eHz0i7QvpfQLX19vjqYjVgkK2NOnT9cff/yh5cuXq1y5cipZsqQkadKkSerUqZNmzZqluXPnPnI5908NDw4OVp48eZzTg4ODVbRo0Rjt169fLw8PD5UpU0aSZLfbJUmNGjVSt27d1K1bt4SsjiQpKir5OyqeDna7g9cb8B/0CyAm+gUQE/0CKV2CAnZQUJDeffddVapUyRlypejfTr/zzjvxvnxWYGCg0qdPr/379zsDdkhIiI4ePaq2bdvGaL9jxw6X27/99psGDhyoxYsXq0iRIglZFQAAAAAATJGggB0SEhLn76wzZsyoe/fid7q1h4eH2rZtq6lTp8rf3185c+bUlClTFBAQoDp16shut+v69evKkCGDvLy8lDdvXpf73x8ILUeOHPLz80vIqgAAAAAAYIoE/XCgcOHC2rJlS6zzvv3223j9/vq+3r17q2XLlho2bJhee+012Ww2LV26VO7u7rp06ZKqVKmi7du3J6RMAAAAAACSTIKOYL/zzjvq2bOnbt68qRo1ashiseinn37Shg0btGbNGk2bNi3ey7LZbBo4cKAGDhwYY16uXLn0559/xnnfihUrPnQ+AAAAAABJJUEBu1atWpoyZYqmTZum7777TpI0ceJEZc6cWSNHjlS9evVMLRIAAAAAgJTusQP24cOHdeHCBRUoUEC7du3SP//8o5s3b8rX11cFChSQ1cpw9QAAAACAp0+8A3ZISIi6du2qX3/9VYZhyGKxqEyZMpo2bZoKFCiQmDUCAAAAAJDixftw88yZM3X06FH16tVLixcv1uDBg/XPP/9o+PDhiVkfAAAAAACpQryPYO/cuVPvvvuu3nzzTUlStWrVlC1bNg0YMED37t2Tj49PohUJAAAAAEBKF+8j2FevXlWJEiVcplWsWFF2u12XLl0yvTAAAAAAAFKTeAfsqKgoeXh4uEzLmDGjJCk8PNzcqgAAAAAASGVMGfLbMAwzFgMAAAAAQKplSsC2WCxmLAYAAAAAgFTrsa6DPXLkSKVPn955+/6R6//7v/9TunTpnNMtFos++ugjk0oEAAAAACDli3fALl++vKSYp4PHNp1TxgEAAAAAT5t4B+yVK1cmZh0AAAAAAKRqpvwGGwAAAACApx0BGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATELABAAAAADABARsAAAAAABMQsAEAAAAAMAEBGwAAAAAAExCwAQAAAAAwAQEbAAAAAAATJHvAdjgcmj17tqpWrarnnntOnTt31rlz5+Js/9dff6lLly6qWLGiKleurN69e+vixYtJWDEAAAAAADEle8CeP3++PvnkE40ZM0Zr1qyRw+FQp06dFBEREaPtjRs31LFjR3l5eWnlypX64IMPdP36dXXq1Enh4eHJUD0AAAAAANGSNWBHRERo2bJl6t27t6pXr67AwEDNmDFDly9f1o4dO2K0//rrr3Xv3j1NnjxZRYoUUcmSJTVlyhSdPHlSBw8eTIY1AAAAAAAgWrIG7OPHj+vu3buqXLmyc5qvr6+KFy+un376KUb7ypUra/78+fLy8nJOs1qjVyEkJCTxCwYAAAAAIA5uyfngly9fliRlz57dZXrWrFmd8x6UK1cu5cqVy2Xa4sWL5eXlpfLlyydeoQAAAAAAPEKyBuzQ0FBJkoeHh8t0T09P3bp165H3X7lypVatWqVhw4bJ39//iWpxc0v2n6NLkmy2lFEHEg/P8eNjm6V9PMePj22W9vEcPz62WdrHc4yULlkD9v1TvSMiIlxO+w4PD5e3t3ec9zMMQ7NmzdKCBQv0zjvvqF27dk9Uh9VqUaZM6Z5oGUB8+frG/doGnlb0CyAm+gUQE/0CKV2yBuz7p4YHBwcrT548zunBwcEqWrRorPeJjIzU0KFDtXXrVg0dOlQdOnR44jocDkMhIfeeeDlmsNmsvHGkcSEhobLbHcldRqpCv0j76BePj36R9tEvHh/9Iu1LKf3C19ebo+mIVbIG7MDAQKVPn1779+93BuyQkBAdPXpUbdu2jfU+gwYN0ldffaVp06apYcOGptUSFZX8HRVPB7vdwesN+A/6BRAT/QKIiX6BlC5ZA7aHh4fatm2rqVOnyt/fXzlz5tSUKVMUEBCgOnXqyG636/r168qQIYO8vLy0YcMGbd++XYMGDVKFChV09epV57LutwEAAAAAIDkk+3kNvXv3VsuWLTVs2DC99tprstlsWrp0qdzd3XXp0iVVqVJF27dvlyRt3bpVkjR58mRVqVLF5d/9NgAAAAAAJIdkPYItSTabTQMHDtTAgQNjzMuVK5f+/PNP5+1ly5YlZWkAAAAAAMRbsh/BBgAAAAAgLSBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACZI9oDtcDg0e/ZsVa1aVc8995w6d+6sc+fOxdn+xo0b6t+/v8qXL68KFSpo1KhRCg0NTcKKAQAAAACIKdkD9vz58/XJJ59ozJgxWrNmjRwOhzp16qSIiIhY2/fu3VtnzpzR8uXLNWvWLH333XcaOXJk0hYNAAAAAMB/JGvAjoiI0LJly9S7d29Vr15dgYGBmjFjhi5fvqwdO3bEaH/o0CEdOHBAkyZNUokSJVS5cmWNHj1amzdv1pUrV5JhDQAAAAAAiJasAfv48eO6e/euKleu7Jzm6+ur4sWL66efforR/ueff9YzzzyjggULOqdVqFBBFotFv/zyS5LUDAAAAABAbJI1YF++fFmSlD17dpfpWbNmdc570JUrV2K09fDwkJ+fny5dupR4hQIAAAAA8Ahuyfng9wcn8/DwcJnu6empW7duxdr+v23vtw8PD09wHVarRf7+6RJ8fzNZLNH/j+xcWVF2R/IW8x+eHjZJUvY2/yfDHpXM1cRkcY9+bbxXrZeiHCmvPjdrdHfLmNFbhpHMxaQyKblfSCm7b9Av0i76xZNJyX2DfpFw9IsnQ7+IP6vVktwlIIVK1oDt5eUlKfq32Pf/lqTw8HB5e3vH2j62wc/Cw8Pl4+OT4DosFotstpTVSfwyeCZ3CXGypcuY3CU8VEavDMldwkNZrck+tmCqlZL7hZSy+wb9Iu2iXzyZlNw36BcJR794MvQLIOGS9RV6/3Tv4OBgl+nBwcHKli1bjPYBAQEx2kZEROjmzZvKmjVr4hUKAAAAAMAjJGvADgwMVPr06bV//37ntJCQEB09elTly5eP0b58+fK6fPmyzpw545x24MABSVLZsmUTv2AAAAAAAOKQrKeIe3h4qG3btpo6dar8/f2VM2dOTZkyRQEBAapTp47sdruuX7+uDBkyyMvLS88++6yef/559evXTyNHjtS9e/c0fPhwNW3aNNYj3gAAAAAAJBWLYSTvMAF2u13Tp0/Xhg0bFBYWpvLly2v48OHKlSuXzp8/r5dfflkTJkxQ8+bNJUn//vuvRo0apd27d8vT01P16tXT0KFD5emZsn9rAwAAAABI25I9YAMAAAAAkBYwDB8AAAAAACYgYAMAAAAAYAICNgAAAAAAJiBgAwAAAABgAgI2AAAAAAAmIGADAAAAAGACAjYAAAAAACYgYKcgn376qbZu3RrrvL1796pZs2Zq3LixunXrplu3bsVoExoaqjJlyqhJkybOf3a7PUa7b7/9Vh9++OFj1TZnzhzNmTPnse7zXytXrlT9+vVVp04dffrpp87phw4dUuvWrdWwYUO9++67ioiIcLnfkCFDtGHDBuft+GyLlOb8+fOqWbPmY93nYa8HJEx8tunRo0dVsmTJOOdfuXJF7dq1U/369dW+fXv9+++/kqSIiAiNHTtWTZo0UcOGDfXDDz+43O/EiRNq2LCh83ZkZKSGDh2qRo0aqXHjxtqyZcsTrFnS+W9/fJTbt2+re/fuiVjR0439RuLsN+7cuaOWLVuqSZMm+uuvv55oHeIrIdurXbt2iVQNHpTa9x1r165VjRo1NG7cuEe2NUvRokUfq31C3mOAlIqAnYIcOnQoxocESbLb7Ro8eLCmTZumLVu2qFChQlq6dGmMdn/88YcqVaqkzZs3O//ZbLZY2925cydR1iEuR48e1Weffab169drw4YNWrVqlU6ePKk7d+6oV69eGj16tLZt2yYpekcgRe+MunXrpqCgIOdy4rst0oK4Xg9IuEdt09DQUI0ePVqRkZFxthk1apSaN2+uoKAgvfLKK84PLEuWLNGNGze0adMmzZw5U0OGDJHD4ZAkbdiwQW+//bZCQ0Ody/nss88UGRmprVu36qOPPtLYsWOTvF8mhVu3bun48ePJXUaaxX4jcfYbx44dk81m0+bNm1W4cOHEW8kndODAgeQu4amQ2vcdW7Zs0ejRo/X+++8/zmonqeR4jwESi1tyF5DWzZo1S9u2bVOGDBlUsGBB5c6dWytXrlTt2rX1+++/y9vbW1OnTtXp06f17bffat++fcqcObNeeukl5zJsNpu++eYbubu7KyIiQleuXIn1m8Hff/9dV65cUatWrWSz2TRgwACVK1fOpc2ff/6pNWvWSJICAgJ0+fJlSVKvXr0kRX8b3rNnT0nS5MmTZRiG8uXLp/z58+vw4cNq3bq17ty5o9atW6tDhw6SpIULF+rzzz+XzWbTiy++qIEDB8b4gLZz507VrVtXPj4+kqS6desqKChIhQsX1nPPPafAwEBJ0rBhwxQVFSVJ2rx5s15++WX5+fk99rZ40KBBg/Tnn39Kkm7evCnDMPT999/ryJEjGj9+vEJDQ5UhQwaNGDFCBQsWVLt27eTr66uTJ09q4sSJunHjhmbOnCmHw6HcuXNr9OjRypIlS5yP980332ju3LmyWCzy8/PTlClTJEnh4eHq37+/Tpw4ITc3N82ePVu5c+fWr7/+qnHjxiksLEz+/v4aPXq0zp49G+frAa7M6GP3TZw4UR06dNChQ4difazIyEjt379fs2bNkiQ1bdpU48ePV2RkpIKCgjRlyhRZLBYVLlxYy5cvl2EYunnzpnbt2qXp06dr8ODBzmW98cYbat26tSQpODhY7u7ucnd3j3M9Dx48qFGjRjlv//333xo5cqSaN2+uqVOnat++fYqKilK9evXUo0cP7d+/36UPjxs3TsOGDdOff/4pi8Wit99+W02bNo3z8cLCwjRo0CCdPXtWFotFr776qtq0aSNJ+v7777V69Wpdu3ZNrVq1Uvfu3eVwODR+/Hj9+OOPslqteuWVV9SlSxeNHj1awcHB6tatmxYuXBj3Ewkn9hvRkmu/8e+//+q9997TtWvX1LlzZy1cuDDWPta4cWNNmTJFgYGBeu+99xQWFqbp06frzJkz6tevX5xnejgcDo0aNUqHDh2SzWZTzZo1ndvy999/V5s2bRQcHKyqVas6+3xs22vs2LGSpObNmz/WWSWI9rTsO+bOnavff/9do0eP1pAhQ5QtW7YYn31Onjypzz//XHPnztWlS5dUvXp1BQUFqUCBAhowYIAaNGgQ51l4v/76q8aOHSvDMOTp6amxY8eqQIECkuR8nYeHh2vSpEkqXbq0Tp06peHDh+vmzZvy8fHR+++/L09PT5f3mFatWsXjGQRSMAOJ5ttvvzVatWplhIaGGvfu3TOaN29uzJ492yhSpIixdu1awzAMY8WKFUbnzp0NwzCMwYMHG+vXr49zeUePHjUqVapkVK1a1bh48WKM+R9++KGxaNEiw+FwGEeOHDFefPFF4/r16zHazZ4925g9e3aMvw3DMNq2bWvs27fP2Ldvn1GmTBnj5s2bznavvPKKcffuXeP27dtG7dq1jaNHjxq7du0yWrRoYdy7d8+IjIw0unXrZqxatSrGY/7f//2f8dlnnzlvf/bZZ8awYcOMRYsWGYMGDTJ69uxpNGrUyBgxYoQRFhbmct/YtsujtkVsQkJCjEaNGhm7d+82IiIijMaNGxvnzp0zDMMwfv75Z6NZs2bObTB9+nTDMAzj2rVrxosvvmicPXvWMAzD+OCDD4xevXo99HGaNGliHD9+3DAMw/joo4+MXbt2GefOnTOKFi1qHDx40DAMwxg/frwxceJEIzw83Khevbpx6NAhwzAMY/v27Ubz5s3jXG+4MrOPff3118agQYMMwzCMIkWKxNomODjYqFq1qsu0qlWrGpcvXzZKlSplrFq1ymjevLnRqlUrY+/evS7tzp07Z9SoUSPGMgcPHmyUKFHCmDVrVrzXe82aNUbbtm2NyMhIY82aNcaYMWMMh8NhREREGJ06dTK+/fbbGH140qRJxqhRowzDMIx///3XqFmzpnHs2LE4H+Orr74yevbsaRiGYVy/ft0YMGCAs94uXboYdrvduHbtmlG6dGnj9u3bxqpVq4xu3boZkZGRxr1794wWLVoYO3fujHO9ETv2G/+TnPuNffv2GW3btjUMw4izj02dOtVYtmyZYRiG0bhxY6NmzZqGYUQ/P3PmzIlz2ceOHXPub8LCwox3333XuHfvnjF79myjadOmRmhoqBEaGmpUqVLFOHHixEO3V1zvVXi4p23fcb+PxvXZ5/bt20blypUNu91urF+/3qhcubKxZs0aw263G9WrVzdCQ0PjXHb37t2Nb775xjAMw9i2bZuxYcMG57bYtm2bYRjRn4Xuf3Zq0aKFsX37dsMwDOPQoUNG9erVjfDw8BjvK0BqxiniiWjPnj1q1KiRvLy85O3trVdeeUWS5O7urubNm0uSmjVrpp9++ileyytWrJj27t2rbt26qV+/fjHmd+jQQV26dJHFYlGJEiVUqlQpHTx4MMH1FyhQQBkzZnTerl+/vnx8fJQ+fXrVqFFDBw4c0L59+9SoUSN5e3vLzc1NLVq00N69e2MsyzCMGNMsFovsdru+++47DRw4UJs2bVJYWJgWL178yNoetS3+y263q1+/fmratKmqVKmiU6dO6ezZs+rRo4eaNGmi0aNH6+rVq85TwJ5//nlJ0uHDh1W6dGnlzp1bkvTqq69q3759D32sWrVqqWvXrhozZowKFizo/LY7a9asKlOmjCSpSJEiunnzpk6fPi1fX18999xzkqK38dmzZ3X79u1HrhPM62NXr17VggUL9H//938PbXf/tL3/slqtstvtOn/+vNatW6fRo0drwIAB8XoeJ06cqO+//15ffvlljN/exWb//v1atmyZZs2aJTc3N+3Zs0e7du1S06ZN1bJlS505c0YnTpyQ5NqH9+3b5zwq4O/vr5dffvmhp5eWKlVKR44c0dtvv60tW7a4HEGpVauWrFarMmfOLH9/f926dUv79+9XixYt5ObmJm9vbzVu3DjW9wI8HPuN/0nu/cZ9cfWxl156ST/++KPOnTunHDly6JlnntHp06f1/fffq0aNGnEuL0+ePIqIiNAbb7yhjz76SP369ZO3t7ckqVq1avLy8pKXl5fy5s2rGzduxHt7If6exn2HpDg/+3h4eKhIkSI6cuSI9u3bpw4dOujAgQM6cuSIihYtKi8vrziXWbNmTQ0bNkzvv/++PDw8nNtSkurUqSMp+jPPjRs3dPfuXZ05c0b169eXJD333HPKmDGj/vnnn3jVD6QWnCKeiKxWa6xvqhaLRRaLRVL0m67VGvN7jiZNmjj/Xr16tfbt2+c8Padp06bO044ftHbtWlWpUkXZs2eXFP3hxM3NTe+//76OHDkiSc5Tyh6s5cEaH/z90P0d/oPrc9/9Zce2flFRUVq9erXzdJ82bdooW7ZsCg4OdrYJDg5WQECAsmTJotKlSytPnjySoj+MrVq1KsYy77t37168tsV/TZw4URkzZtTbb78tSc7TvTdv3uxcnytXrsjDw0OSnDuT/66fYRgP/Y2VJPXs2VMNGjTQd999pylTpujw4cNq3Lix3Nz+190sFosMw4h1+xmG4TzdEQ9nVh9r27atbt68qTfeeMNl/sqVK10GEVq3bp3u3LmjqKgoubm5KSoqSnfv3pWfn5+yZMmi+vXry2KxKDAwUAEBATp16pRKly4da+2HDx9WpkyZlDt3bvn7+6tatWr6888/VaVKlTjX9+zZsxo8eLAWLFggf39/SdFfHg0aNMj5QebGjRvy8vLS4cOHXfrwf8PKo15n2bJlU1BQkPbs2aPdu3erWbNmzt+7Pngqb1yvZV7HCcN+I+XsN+6Lq495eHjor7/+0u7du1WxYkXdvHlT33//vc6ePasSJUrEuTwfHx9t2rRJ+/fv1w8//KA2bdpo5cqVkhTv/QR968k8bfuO+x722ad69er68ccfdezYMQ0fPlxNmjRRvnz5HjlAa4sWLVS5cmXt2rVLy5cv165du5zvGfdfz/e3aWxfmrGvQFrEEexE9OKLLyooKEjh4eGKiIhQUFCQLBaLIiIi9PXXX0uKHsDixRdflBT9ofX+6K0PDjjj5ubm/O2kJG3bti3Gb+Sk6N9urVixQlL07zOPHj2qsmXLaty4cc5llSpVSjabzflmlilTJucARKdOnXI+Rmx27NihiIgI3bp1Szt37lSlSpVUqVIlbd26VaGhoYqKitL69etVvnx5vfbaa87HfO211/TSSy/pyy+/1N27d3X37l198cUXeumll1SlShUdPXpUFy5ckCR99913Kl68eJw1xHdbPGjt2rU6ePCgy+iZBQoU0K1bt5zfTm/ZskXdunWLcd9nn31Whw8f1rlz5yRFjyRaoUKFhz5eo0aNJEkdO3ZUhw4ddPTo0Tjb3q/j119/lSRt375dAQEBypQpk8vrAbEzq4+1atVKX3/9tfP2/fm+vr4u7dzd3VWhQgXnqK1btmxRhQoV5O7urho1ajgHVjp//rwuXbqk/Pnzx1n7Tz/9pOnTp8swDN2+fVs//PCDypYtG2f7O3fuqHv37ho0aJCKFSvmnF6pUiXnoDehoaHq0KGD9uzZE+P+lSpVcg4Edf36dX399dcP7TtbtmzRyJEj9fLLL2vYsGHy8fHRpUuX4mxfqVIlrV+/XlFRUQoNDdWWLVtUvnx554dJxA/7jZSx33hQXH3MZrOpbNmyWr58uSpWrKhKlSrpgw8+UKVKlR66vJ9//lmdO3dWpUqVNHjwYBUsWFCnTp166OPHtr0kuTwviL+nad/xoId99qlevbo2bNigXLlyKX369MqWLZvWrVun6tWrP3SZnTt31qlTp/T666+rT58+D/3Mkz59euXOndu5vr/++quCg4NVpEgRXstIUziCnYheeukl/f7772rWrJnSpUunTJkyydPTU1L0QFizZ8/WM888o4kTJ0qSqlSpoilTpihdunRq0KCBczkeHh6aPn26c2TJgIAA57eDq1evVnBwsPr06aN+/fpp6NChatiwoaxWqyZPnqz06dPHqKtixYoaOHCgMmXKpKZNm+qbb75RvXr1VKBAgYe+SefMmVOvvfaaQkND1aVLFxUsWFAFCxbUsWPH1LJlS0VFRemFF15Q+/btY9y3VKlSatWqlVq3bq2oqCi1adPG+Q3/2LFj9c477ygiIkJFixbVgAED4qzhYdsiLqNGjXLWfv8b6/un2I4fP15hYWHy8fHR1KlTY9w3S5YsGj16tHr27KmoqCgFBARo/PjxD328/v37q0+fPnJ3d5eXl5dGjhz50PWZMWOGxo0b5xxwZObMmZLifj3gf8zqY49jxIgRGjp0qJYsWaKMGTM6XzcDBgzQ6NGjnZdTGT16tDJkyBDnctq3b6+RI0eqcePGslgsat++vfOnArFZtWqVLly4oKVLl2rRokWSpNq1a6tr1646c+aMmjVrpsjISDVs2FC1atXS/v37Xe7fo0cPjRw5Uo0aNZLdbleXLl3iPEIiRZ/a980336hhw4Zyd3dX3bp1Hzow1KuvvqrTp0+radOmioyMVKNGjVSvXj3Z7XblzJlTr7/+uj755JM4749o7Df+Jzn3Gw9q06ZNrH1Mig4lu3fvVmBgoCIjI3Xz5s1HHvErW7asChQo4DxFuXjx4qpWrZr++OOPWNvXqFEjzu1Vu3ZtvfLKK1q3bp1zMDg82tO073iQh4dHnJ998uXLJ4vFoooVK0qK/mLn+++/V9asWR+6zJ49e2rUqFGaOnWq3NzcNGTIkIe2nzJlikaOHKn58+fL3d1dc+bMkYeHh8t7zP3BEIHUymLEdr4GTPHbb7/pxIkTatWqlQzDUO/evdWiRQt17dr1od/4A4gf+hjSGl7TQOKjnwFITBzBTkT58uXTvHnznKffVa9e/ZGn2iBhJk2apB9//DHG9Pz58zuPCJupf//++vvvv2NMr1ixot577z3THw+xS2t97OzZs87L9fzX0KFDH3nq6eP6+eefNWbMmFjnzZgxw3mpFSSdtPaaTskSc7+R1H0Zjyet9bPEfr0l9WcsILXjCDYAAAAAACZgkDMAAAAAAExAwAYAAAAAwAQEbAAAAAAATEDABgAAAADABARsAAAAAABMQMAGAAAAAMAEBGwAAAAAAExAwAYAAAAAwAQEbAAAAAAATPD/AMjZiieeTlhIAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Extracting the model names and the metric values\n",
    "models = list(by_model_and_context.keys())\n",
    "metrics = list(by_model_and_context[models[0]].keys())\n",
    "\n",
    "# Plotting the bar chart with metric scores on top of each bar\n",
    "fig, ax = plt.subplots(figsize=(10, 4))\n",
    "width = 0.2\n",
    "x = range(len(models))\n",
    "\n",
    "for i, metric in enumerate(metrics):\n",
    "    metric_values = [by_model_and_context[model][metric] for model in models]\n",
    "    ax.bar([pos + width * i for pos in x], metric_values, width, label=metric)\n",
    "    # Displaying the metric scores on top of each bar\n",
    "    for pos, val in zip(x, metric_values):\n",
    "        ax.text(pos + width * i, val, f'{val:.3f}', ha='center', va='bottom', fontsize=9)\n",
    "\n",
    "ax.set_xticks([pos + width for pos in x])\n",
    "ax.set_xticklabels(models, rotation=0, ha='center', fontsize=8)\n",
    "ax.set_ylabel('Performance')\n",
    "ax.set_title('GPT Benchmarks')\n",
    "ax.legend(loc='upper left', bbox_to_anchor=(1, 1))\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "eeb8c5ba-c22f-4fae-b63c-d13f8c23ade7",
   "metadata": {
    "tags": []
   },
   "source": [
    "Our best model is GPT 4 with few shot learning at an f1 score of ~92%. We will see in the [Made With ML course](https://madewithml.com/) how fine-tuning an LLM with a proper training dataset to change the actual weights of the last N layers (as opposed to the hard prompt tuning here) will yield similar/slightly better results to GPT 4 (at a fraction of the model size and inference costs).\n",
    "\n",
    "However, the best system might actually be a combination of using these few-shot hard prompt LLMs alongside fine-tuned LLMs. For example, our fine-tuned LLMs in the course will perform well when the test data is similar to the training data (similar distributions of vocabulary, etc.) but may not perform well on out of distribution. Whereas, these hard prompted LLMs, by themselves or augmented with additional context (ex. arXiv plugins in our case), could be used when our primary fine-tuned model is not so confident."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
